8. Image Segmentation

Introduction and Context

Image segmentation is the process of dividing an image into meaningful, spatially coherent regions. Its purpose is not merely to assign labels to individual pixels, but to identify structures that are useful for interpretation, measurement, recognition, or decision-making. In a medical image, segmentation may separate a tumour from surrounding tissue; in autonomous driving, it may distinguish road, vehicles, pedestrians, and vegetation; in industrial inspection, it may isolate scratches, cracks, or defective components.

Segmentation occupies a central position between low-level image processing and high-level computer vision. Enhancement and restoration improve image quality, whereas segmentation converts image data into regions or objects that can subsequently be measured or classified. The quality of later operations—shape analysis, object recognition, tracking, and diagnosis—often depends more on segmentation accuracy than on the sophistication of the subsequent classifier.

A segmentation result can be represented as a binary mask, a labelled partition, a set of contours, or a probabilistic map. No single method is optimal for every image. The appropriate choice depends on illumination, noise, object appearance, spatial context, computational resources, and the intended meaning of an “object.”

Learning Objectives

After studying this chapter, the reader should be able to:

1. Explain the purpose, terminology, and mathematical formulation of image segmentation.

2. Apply global, adaptive, and multilevel thresholding appropriately.

3. Analyse edge-, region-, clustering-, graph-, and energy-based methods.

4. Distinguish semantic, instance, and panoptic segmentation.

5. Select segmentation methods according to image characteristics and application requirements.

6. Evaluate segmentation quality using suitable region-, boundary-, and object-level measures.

7. Recognise practical, ethical, and operational risks in deploying segmentation systems.

Segmentation as a Partitioning Problem

Let an image domain be denoted by \(D\), containing pixels or voxels, and let \(I(x)\) represent image intensity or a feature vector at location \(x\). A segmentation is a collection of nonempty regions \(R_1,R_2,\ldots,R_n\) satisfying:

\[

D=\bigcup_{i=1}^{n}R_i,

\qquad R_i\cap R_j=\varnothing \quad (i\neq j).

\]

A useful segmentation generally seeks regions that are internally homogeneous with respect to a selected property, while adjacent regions differ significantly. Homogeneity may concern intensity, colour, texture, motion, depth, or learned features. Connectivity is also important: two pixels with similar intensity may belong to different objects if they are spatially separated.

The term foreground refers to the region or objects of interest, while background refers to the remaining image content. A binary segmentation produces a mask \(M(x)\in\{0,1\}\); a multiclass segmentation assigns each pixel a class label. A hard segmentation gives one definite label, whereas a soft segmentation gives class probabilities or confidence values.

Several distinctions must be maintained. A boundary is a local transition between regions; a contour is an explicit curve representing such a boundary. A connected component is a maximal connected set of pixels having a common label. Over-segmentation creates too many regions, while under-segmentation merges distinct structures. These errors have different consequences: over-segmentation may be corrected by merging, but under-segmentation can permanently obscure object identity.

Thresholding: Separating Regions by Measurement

Thresholding is the simplest segmentation strategy. For a grayscale image, a binary mask can be written as

\[

M(x)=

\begin{cases}

1,& I(x)\geq T,\\

0,& I(x)<T,

\end{cases}

\]

where \(T\) is a threshold. Thresholding is effective when foreground and background have distinguishable intensity distributions.

Global and Multilevel Thresholding

A global threshold uses one value for the entire image. It is appropriate when illumination is uniform and the object has relatively stable contrast. A histogram-based method such as Otsu’s method chooses a threshold that maximises between-class variance, or equivalently reduces within-class variance, under a two-class model. Otsu thresholding is attractive because it requires no manually specified threshold, but it can fail when classes have unequal areas, overlapping intensities, or strong illumination gradients.

Multilevel thresholding uses several thresholds to create more than two intensity classes. It is useful in images containing background, object material, and bright or dark substructures. However, increasing the number of classes also increases sensitivity to noise and histogram irregularities.

Adaptive Thresholding

When lighting varies spatially, a local threshold is calculated from a neighbourhood around each pixel. Common choices use the local mean, a Gaussian-weighted mean, or a statistic such as the local median. Adaptive thresholding can recover text under shadows or objects under nonuniform illumination, but neighbourhood size is critical. A window that is too small responds to noise; one that is too large behaves almost like a global threshold.

Thresholding should rarely be treated as the entire solution. Smoothing before thresholding may suppress noise, and morphological opening or closing can remove isolated pixels and fill small gaps. Connected-component analysis can then eliminate regions that are too small or geometrically implausible.

Edge-Based Segmentation

Edges occur where image intensity or another image property changes rapidly. The gradient magnitude,

\[

\|\nabla I\|=\sqrt{I_x^2+I_y^2},

\]

provides a measure of local change. Edge-based segmentation usually involves smoothing, gradient computation, non-maximum suppression, thresholding, and edge linking. The Canny detector is a widely used framework because it combines noise reduction with good localisation and relatively thin edges.

Edges are useful when region interiors are similar but their boundaries are sharp. They are especially valuable for measuring contours, detecting manufactured parts, and locating geometric structures. Nevertheless, an edge map is not automatically a segmentation. It may contain fragmented edges, texture edges, shadows, and irrelevant detail. A complete method must link contours, close gaps, and determine which closed boundaries correspond to objects.

The choice of gradient operator and smoothing scale creates a trade-off. Strong smoothing reduces noise but can displace or erase narrow structures. Weak smoothing preserves detail but produces many false edges. Multi-scale approaches analyse the image at several resolutions and select structures that remain stable across scales.

Region-Based Methods

Region growing begins with one or more seed pixels and adds neighbouring pixels whose properties satisfy a similarity criterion. A seed may be selected manually, by an automatic detector, or from intensity extrema. Region growing is intuitive and can produce connected regions, but its outcome depends on seed location, similarity thresholds, and the order in which pixels are considered.

Region splitting starts with the whole image. If a region fails a homogeneity test, it is divided, often into four quadrants in a quadtree. Region merging combines adjacent regions when their statistics are sufficiently similar. Split-and-merge methods use both ideas and can adapt to spatial structure, although the homogeneity criterion strongly influences the result.

Watershed segmentation interprets an image, often a gradient image, as a topographic surface. Water would collect in catchment basins, while watershed lines form boundaries between basins. This method is effective for separating touching objects, particularly when reliable markers are available. Applied without markers, however, it is highly prone to over-segmentation because small intensity variations create many basins. Marker-controlled watershed reduces this problem by specifying likely object interiors and background regions.

Clustering and Statistical Segmentation

Clustering treats pixels or local descriptors as samples in a feature space. In k-means clustering, each sample is assigned to the nearest of \(k\) centroids and centroids are repeatedly updated to minimise within-cluster squared distance. Features may include colour channels, texture measures, coordinates, or learned embeddings. Spatial coordinates can encourage compact regions, but excessive spatial weighting may incorrectly divide a single object.

Fuzzy c-means permits partial membership. A pixel may belong to several clusters with different degrees of membership, which is useful when boundaries are uncertain or intensities overlap. The resulting membership values can be converted into a hard mask, but retaining them may support uncertainty-aware decisions.

Gaussian mixture models represent feature distributions probabilistically and can be estimated using expectation-maximisation. Unlike k-means, they can model clusters with different variances and provide posterior probabilities. Their assumptions may still be inadequate for highly non-Gaussian or spatially dependent image data.

Clustering methods are unsupervised, but the number of clusters and feature design remain important modelling decisions. Purely intensity-based clustering often ignores object connectivity. Spatial regularisation, superpixels, or subsequent morphological processing can improve coherence.

Graph-Based and Energy-Based Formulations

Graph-based segmentation represents pixels, superpixels, or regions as nodes in a graph. Edges connect neighbouring or otherwise related nodes and carry weights reflecting similarity. A segmentation can then be formulated as a partition that balances data evidence with spatial smoothness.

In a common binary energy model,

\[

E(L)=\sum_{p}D_p(L_p)+\lambda\sum_{(p,q)}V_{pq}(L_p,L_q),

\]

where \(L_p\) is the label of pixel \(p\), \(D_p\) is a data cost measuring how well the label explains the pixel, and \(V_{pq}\) penalises undesirable discontinuities between neighbouring pixels. The parameter \(\lambda\) controls the balance between fidelity to image evidence and spatial regularity.

Graph cuts can efficiently minimise important classes of binary or multi-label energies. In interactive segmentation, user-provided foreground and background seeds guide the data terms, while graph connectivity propagates the decision through similar pixels. Random walker methods assign labels according to probabilities of reaching seeded nodes through random walks. Normalised cuts formulate segmentation through graph partitioning and seek groups with strong internal similarity and weak external similarity.

Active contours, or snakes, are energy-based curves that move toward object boundaries. Their energy may include image-gradient terms, curve smoothness, and region-based statistics. Parametric snakes require an initial contour and may struggle with topology changes. Level-set methods represent the contour implicitly as a level set of a higher-dimensional function, allowing natural splitting and merging. They are powerful but computationally more demanding and sensitive to model design.

Deep Learning and Modern Segmentation Systems

Convolutional neural networks transformed segmentation by learning hierarchical image features directly from data. Fully convolutional networks replace classification layers with dense prediction operations. Encoder–decoder architectures reduce spatial resolution to obtain semantic context and then recover detailed output through upsampling and skip connections. U-Net is particularly influential in biomedical imaging because its contracting path captures context while its skip connections preserve fine spatial detail.

Semantic segmentation assigns a class to every pixel. All objects of the same class receive the same label; two neighbouring cars, for example, are not separated. Instance segmentation additionally distinguishes individual objects within a class. Panoptic segmentation combines semantic segmentation for background or “stuff” classes with instance segmentation for countable “things.”

Modern systems often use attention mechanisms, transformer-based encoders, promptable segmentation, and foundation models. These systems can transfer knowledge across datasets, but performance depends on domain similarity, annotation quality, and the reliability of adaptation. A visually impressive mask does not guarantee suitability for safety-critical or clinical use.

Selecting a Segmentation Strategy

Method selection should begin with the application rather than with the popularity of an algorithm. Thresholding is attractive for controlled industrial imagery with stable contrast. Edge methods are suitable when boundaries are sharply defined and region interiors are less informative. Region methods are useful when connectivity and homogeneity are central. Clustering is valuable for exploratory analysis or when labelled data are limited. Graph and energy methods are appropriate when spatial coherence and user interaction matter. Deep learning is preferred when complex appearance variation exists and sufficient representative training data are available.

A practical workflow commonly includes image inspection, calibration and normalisation, noise reduction, a candidate segmentation method, post-processing, validation, and failure analysis. Post-processing may include morphological operations, hole filling, connected-component filtering, contour smoothing, or shape-constrained refinement. It should not conceal systematic failures by deleting difficult cases.

Evaluating Segmentation Quality

Evaluation must match the application. For binary masks, the Dice coefficient is

\[

\mathrm{Dice}=\frac{2|P\cap G|}{|P|+|G|},

\]

where \(P\) is the predicted region and \(G\) is the ground-truth region. Intersection over Union (IoU), also called the Jaccard index, is

\[

\mathrm{IoU}=\frac{|P\cap G|}{|P\cup G|}.

\]

Dice is often more sensitive to overlap when regions are small, whereas IoU has a direct union-based interpretation. Pixel accuracy can be misleading when the background dominates. Precision measures the proportion of predicted foreground that is correct; recall measures the proportion of actual foreground that is recovered. Their relative importance depends on whether false alarms or missed regions are more costly.

Boundary quality requires additional measures. Boundary precision and recall compare predicted and reference contours within a tolerance. Hausdorff distance captures the largest boundary discrepancy, while average symmetric surface distance gives a less extreme measure of contour separation. For instance segmentation, object-level matching, counting accuracy, and average precision are important. Panoptic quality evaluates recognition and segmentation together by matching predicted and reference instances.

Evaluation should use a representative test set separated from training data. Confidence intervals, subgroup analysis, calibration, and visual inspection of failure cases are valuable. In medical applications, inter-annotator disagreement should be considered because a “ground truth” boundary may itself be uncertain.

Applied Case Study: Segmentation of Cell Nuclei

Background

Consider an illustrative research laboratory developing an automated system to segment cell nuclei in fluorescence microscopy images. The system will estimate nuclear count, area, and shape for downstream analysis of cell growth.

Situation and Problem

Images contain uneven illumination, sensor noise, overlapping nuclei, and occasional debris. A single global threshold works on some fields but fails on dim regions. Overlapping nuclei are merged, while bright debris is sometimes identified as a nucleus. Counting errors are more damaging than small boundary deviations.

Analysis

The problem has both appearance and topology components. Intensity thresholding can produce an initial foreground mask, but spatially varying illumination requires background correction or adaptive thresholding. Since the objects are approximately compact and separated by narrow intensity valleys, marker-controlled watershed is a plausible refinement. If many annotated images are available, an instance-segmentation network may provide a stronger solution, but annotation effort and domain shift must be assessed.

Approach and Solution

The proposed pipeline first estimates slowly varying background illumination and subtracts it from each image. A denoising filter is applied without excessively blurring nuclear boundaries. Adaptive thresholding creates a preliminary mask, followed by morphological opening to remove isolated noise. Distance transform peaks within the foreground mask provide candidate nuclear markers. A marker-controlled watershed separates touching nuclei. Components outside plausible area and shape ranges are flagged rather than automatically discarded. A validation set is evaluated using Dice and IoU for masks, object-level recall and precision for nuclei, and counting error per image.

Outcome

The approach is expected to improve robustness over a single global threshold because it addresses illumination variation and touching objects separately. However, dim nuclei and irregularly shaped nuclei remain difficult. The laboratory therefore retains an image-review stage for uncertain cases and records performance separately for different microscopes and staining conditions.

Learning Points

The case illustrates that segmentation is a pipeline rather than a single algorithm. It also demonstrates the importance of selecting metrics according to operational goals: a method with excellent pixel overlap may still merge two nuclei and produce an unacceptable count. Domain-specific validation and uncertainty handling are essential.

Limitations, Risks, and Professional Considerations

Segmentation methods can fail because of noise, weak boundaries, shadows, occlusion, texture, scale variation, and domain shift. A model trained on one camera, hospital, or staining protocol may degrade on another. Annotation errors can teach the system inconsistent boundaries. Class imbalance can make accuracy appear high even when the object of interest is poorly detected.

Common mistakes include choosing a threshold without inspecting intensity distributions, applying morphological operations with unsuitable structuring-element sizes, assuming that all edges represent object boundaries, ignoring connectivity, and reporting only one aggregate metric. Another error is tuning a method repeatedly on the test set, which converts the test set into an implicit training resource.

In professional settings, segmentation outputs may influence medical treatment, manufacturing release, biometric decisions, or public-safety systems. Such use requires traceability, validation, access control, audit logs, and human oversight proportionate to risk. Training images may contain personal or sensitive information and must be handled under applicable privacy and data-governance requirements. Automated masks should not be presented as objective truth when uncertainty and annotation disagreement are substantial.

Emerging Directions and Best Practices

Current research is moving toward self-supervised representation learning, weakly supervised and semi-supervised segmentation, interactive foundation models, uncertainty estimation, and domain adaptation. These developments aim to reduce annotation costs and improve transfer across imaging conditions. Three-dimensional and temporal segmentation are also increasingly important, requiring consistency across slices or frames rather than independent two-dimensional predictions.

Reliable practice follows several principles:

• define the object and the required level of detail before selecting an algorithm;
• preserve a representative, independently held-out test set;
• combine region, boundary, and object-level evaluation;
• inspect failures by subgroup, acquisition condition, and object size;
• document preprocessing, parameters, training data, and post-processing;
• quantify uncertainty and route ambiguous cases for review;
• prefer the simplest method that meets the operational requirement.

Segmentation is therefore best understood as an interaction between image evidence, spatial structure, prior knowledge, and application objectives. Technical sophistication is valuable only when it produces dependable, interpretable, and fit-for-purpose results.

Chapter Summary

Image segmentation partitions an image into meaningful regions or objects and provides the foundation for measurement and recognition. Thresholding uses intensity distributions; edge methods detect transitions; region methods exploit connected homogeneity; clustering groups pixels in feature space; and graph or energy methods balance data evidence with spatial coherence. Active contours and level sets formulate segmentation through evolving boundaries. Deep learning extends these ideas through learned hierarchical features and supports semantic, instance, and panoptic segmentation.

No universal method exists. Selection depends on illumination, noise, object appearance, spatial relationships, available annotations, and application risk. Dice, IoU, precision, recall, boundary distances, and object-level measures provide complementary evidence of quality. Robust deployment requires representative validation, failure analysis, uncertainty management, privacy protection, and human oversight where consequences are significant.

Key Terms

Segmentation: Partitioning an image into meaningful regions or labelled objects.

Foreground: Region or objects selected as the primary subject of analysis.

Thresholding: Assigning labels according to whether image measurements exceed selected thresholds.

Adaptive thresholding: Thresholding in which the decision value varies spatially.

Edge: A location of rapid change in intensity, colour, or another image property.

Contour: A curve representing a detected or estimated object boundary.

Region growing: Expanding a seed region by adding neighbouring pixels satisfying a similarity rule.

Watershed: A topographic segmentation method that divides an image into catchment basins.

Clustering: Grouping pixels or features according to similarity without necessarily using class labels.

k-means: An iterative clustering method that minimises within-cluster squared distance.

Graph cut: A graph-optimisation approach that partitions nodes by minimising an energy function.

Active contour: An evolving curve guided by image evidence and regularisation terms.

Level set: An implicit representation of a contour as a level curve of a higher-dimensional function.

Semantic segmentation: Assigning a class label to every pixel without distinguishing object instances.

Instance segmentation: Assigning pixels to classes while separating individual objects.

Panoptic segmentation: A unified representation of semantic and instance segmentation.

Dice coefficient: An overlap measure equal to twice intersection divided by the sum of predicted and reference areas.

Intersection over Union: Overlap divided by the union of predicted and reference regions.

Over-segmentation: Splitting one meaningful object or region into multiple predicted regions.

Under-segmentation: Merging distinct meaningful objects or regions into one predicted region.

Review Questions

Short Answer Questions

1. What is the difference between a boundary and a region in image segmentation?

2. Why can global thresholding fail under spatially varying illumination?

3. State one strength and one limitation of watershed segmentation.

4. Distinguish semantic segmentation from instance segmentation.

5. Why can pixel accuracy be misleading in a highly imbalanced image?

Long Answer Questions

1. Explain global, adaptive, and multilevel thresholding, including the conditions under which each is appropriate.

2. Compare edge-based, region-growing, and split-and-merge segmentation methods.

3. Describe the energy formulation of graph-based segmentation and explain the roles of data and smoothness terms.

4. Discuss semantic, instance, and panoptic segmentation with suitable application examples.

5. Develop a complete segmentation workflow for an industrial inspection problem and justify each stage.

Analytical / Application Questions

1. An image contains two touching bright objects on a dark background with uneven illumination. Propose a segmentation pipeline and explain why each step is needed.

2. A model obtains high pixel accuracy but low object recall on small lesions. Which evaluation measures and diagnostic analyses should be added, and why?

3. A segmentation model performs well in the laboratory but poorly after deployment at another site. Analyse possible causes and propose a validation and adaptation strategy.

22. Multiple Choice Questions with Answers

Question 1

Which condition most strongly favours the use of global thresholding?

A. Severely nonuniform illumination

B. Uniform illumination with separable foreground and background intensities

C. Objects with no intensity contrast

D. Images in which only boundary gradients are available

Correct Answer: B. Uniform illumination with separable foreground and background intensities

Question 2

What is the principal purpose of marker-controlled watershed segmentation?

A. To eliminate all image gradients

B. To convert a colour image into grayscale

C. To reduce over-segmentation and separate touching objects using markers

D. To learn object features without training data

Correct Answer: C. To reduce over-segmentation and separate touching objects using markers

Question 3

In an energy-based segmentation model, the smoothness term generally encourages neighbouring pixels to:

A. Have unrelated labels regardless of image evidence

B. Receive compatible labels unless a boundary is supported

C. Have identical intensity values

D. Belong to different classes

Correct Answer: B. Receive compatible labels unless a boundary is supported

Question 4

What distinguishes instance segmentation from semantic segmentation?

A. Instance segmentation assigns labels only to background pixels

B. Semantic segmentation always requires manual seeds

C. Instance segmentation separates individual objects of the same class

D. Semantic segmentation cannot use deep learning

Correct Answer: C. Instance segmentation separates individual objects of the same class

Question 5

Two predicted and reference regions have an intersection of 60 pixels and a union of 100 pixels. What is their IoU?

A. 0.40

B. 0.60

C. 1.67

D. 0.30

Correct Answer: B. 0.60

Question 6

Why might k-means clustering alone produce fragmented image regions?

A. It always uses boundary gradients

B. It ignores feature similarity

C. It may group pixels by feature similarity without enforcing spatial connectivity

D. It cannot assign pixels to clusters

Correct Answer: C. It may group pixels by feature similarity without enforcing spatial connectivity

Question 7

In the illustrative nuclei case, why was object-level recall important in addition to Dice?

A. Because the application required accurate counting and missed nuclei were consequential

B. Because Dice cannot be computed for binary masks

C. Because recall measures boundary smoothness

D. Because object-level recall eliminates the need for ground truth

Correct Answer: A. Because the application required accurate counting and missed nuclei were consequential

Question 8

Which metric is specifically designed to assess overlap between predicted and reference regions?

A. Dice coefficient

B. Hausdorff distance only

C. Processing latency

D. Number of network parameters

Correct Answer: A. Dice coefficient

Question 9

A segmentation system trained in one hospital fails in another because scanner and acquisition conditions differ. This is primarily an example of:

A. Domain shift

B. Overlapping classes being impossible

C. Perfect calibration

D. Multilevel thresholding

Correct Answer: A. Domain shift

Question 10

Which statement best describes panoptic segmentation?

A. It detects only edges and contours

B. It combines semantic labels for regions with instance identities for countable objects

C. It assigns one label to the entire image

D. It is limited to binary masks

Correct Answer: B. It combines semantic labels for regions with instance identities for countable objects