Segmentation and Mathematical Morphology

Introduction

Image segmentation partitions an image into meaningful regions so that subsequent tasks—measurement, recognition, inspection, tracking, and robotic manipulation—can operate on objects rather than on undifferentiated pixels. In a mechatronic system, segmentation is often the bridge between image acquisition and physical action. A camera may observe a manufactured component, a conveyor, or a robot workspace; segmentation converts intensity or colour measurements into object masks, contours, and regions suitable for geometric estimation and control.

Segmentation is not a single algorithm. It is a family of methods whose suitability depends on illumination, object contrast, texture, noise, scale, occlusion, and computational resources. Thresholding is effective for clearly separated intensity classes, whereas edge methods emphasize boundaries, region methods exploit local similarity, clustering uses feature-space organization, and graph-based methods represent segmentation as an optimization problem. Mathematical morphology provides shape-based operations that clean, connect, separate, and quantify binary or grayscale structures.

A robust vision engineer therefore treats segmentation as a modelling problem. The image formation conditions must be understood, assumptions must be stated, parameters must be validated, and failure modes must be anticipated.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Explain the purpose and assumptions of major segmentation families.
2. Apply global, adaptive, and colour-based thresholding.
3. Detect boundaries using gradient and edge-linking methods.
4. Compare region growing, split-and-merge, clustering, watershed, and graph-based segmentation.
5. Construct simple background models for moving-object extraction.
6. Use erosion, dilation, opening, closing, reconstruction, and connected-component analysis.
7. Select structuring elements and parameters appropriate to object geometry.
8. Analyse segmentation robustness under illumination variation, texture, noise, and occlusion.
9. Evaluate segmentation quantitatively and design a practical inspection pipeline.

1. Segmentation as an Inference Problem

Let an image be represented by a function I(x,y) defined on pixel coordinates (x,y). A segmentation assigns each pixel to a region or class. In binary segmentation, the result is a mask M(x,y) with values 1 for foreground and 0 for background. In multiclass segmentation, a label function L(x,y) assigns one of several class labels.

The segmentation problem is underdetermined: different objects can produce similar pixel values, and the same object can appear differently because of shadows, reflections, lens effects, or pose. Practical methods introduce assumptions such as:

• foreground and background have separable intensity distributions;
• neighbouring pixels belonging to one object are locally similar;
• object boundaries generate strong intensity gradients;
• objects have characteristic colour, texture, or shape;
• the background is stable over time.

No method is universally best. A useful design question is: which image property remains most stable in the application? If intensity is stable, thresholding may suffice. If intensity varies but boundaries remain visible, edge or region methods may be preferable. If the scene is static and only motion changes, background modelling can be highly effective.

2. Threshold-Based Segmentation

2.1 Global thresholding

For a grayscale image, a global threshold creates a binary mask:

Here T is a scalar threshold in the intensity range. The inequality may be reversed when the object is darker than the background. Global thresholding assumes that one threshold separates the relevant classes throughout the image. It is computationally inexpensive and easy to implement, making it valuable in controlled industrial lighting.

A threshold can be selected manually, from a histogram, or by an automatic method. Otsu’s method selects the threshold that maximizes between-class variance. For threshold t, foreground and background probabilities are ω₀(t) and ω₁(t), and their means are μ₀(t) and μ₁(t). The total mean is μ_T. The between-class variance is

The selected threshold maximizes σ_B²(t). This method is most appropriate when the histogram is approximately bimodal and the class distributions have adequate separation. It can fail when illumination produces a broad or multimodal background distribution.

2.2 Adaptive thresholding

When illumination varies spatially, a threshold should depend on location. A common local rule is

where μ_N(x,y) is the mean intensity in a neighbourhood N around the pixel and C is a chosen offset. Median or Gaussian-weighted local statistics may replace the mean. The neighbourhood must be large enough to represent illumination but small enough to follow local changes. Excessively small windows respond to texture and noise; excessively large windows behave almost like global thresholding.

2.3 Colour and multispectral thresholding

Grayscale conversion can destroy useful information. Colour segmentation may use RGB, HSV, HSL, Lab, or normalized chromaticity. HSV separates hue and saturation from brightness, often making colour selection less sensitive to moderate intensity changes. However, hue is unstable when saturation is low, and specular highlights can distort all colour spaces.

A colour rule may be expressed as a set-membership condition:

where f maps the pixel to a feature vector and R is an accepted region in feature space. The region may be a box, ellipse, or learned classifier. Sensor calibration and controlled illumination remain important.

Worked Example 1: Global threshold

Suppose a grayscale inspection image contains a bright component with intensities mostly between 180 and 230, while the background is between 20 and 90. Selecting T = 140 gives foreground pixels when I(x,y) ≥ 140. A pixel with intensity 205 is labelled foreground; a pixel with intensity 75 is labelled background. The rule is reliable only if shadows or bright background reflections do not cross 140. The numerical operation is simple, but validation must include images captured at different times and positions.

3. Edge-Based Segmentation

Edges are locations where image intensity changes rapidly. For a differentiable image, the gradient is

and its magnitude is

The gradient direction indicates the direction of greatest increase. In digital images, derivatives are approximated using kernels such as Sobel or Scharr operators. Noise amplifies under differentiation, so smoothing is commonly performed first.

The Canny edge detector illustrates a principled pipeline: Gaussian smoothing, gradient calculation, non-maximum suppression, double thresholding, and hysteresis edge linking. The Gaussian smoothing kernel is

where σ is the standard deviation in pixels. Larger σ suppresses more fine detail but can displace or merge nearby edges. Canny’s high and low thresholds should be selected relative to image noise and expected boundary contrast.

Edges are not automatically closed object boundaries. Gaps, texture edges, shadows, and multiple contours can produce incomplete or misleading regions. Edge methods therefore often require contour linking, filling, morphology, or geometric constraints.

4. Region-Based Segmentation

4.1 Region growing

Region growing begins with one or more seed pixels and adds neighbouring pixels that satisfy a similarity criterion. For intensity similarity, a candidate pixel may be accepted when

where p is the candidate pixel, μ_R is the current region mean, and τ is a tolerance. The region mean may be updated after each accepted pixel. Connectivity—4-neighbour or 8-neighbour in a rectangular image—must be specified.

Region growing can produce coherent regions and tolerate weak boundaries, but it is sensitive to seed selection, leakage across gradual boundaries, and accumulated intensity variation. Multiple seeds and stopping rules improve reliability.

4.2 Split-and-merge

Split-and-merge methods begin with a large region, test a homogeneity predicate, split nonhomogeneous regions into subregions, and merge adjacent regions that satisfy the predicate. A quadtree is a common implementation for rectangular images. Homogeneity may involve intensity variance, colour dispersion, or texture. These methods represent spatial structure well but can generate block-like boundaries and require computationally significant region management.

5. Clustering and Statistical Classification

Clustering assigns pixels to groups in a feature space such as intensity, colour, position, or texture. In k-means clustering, the objective is

where C_i is cluster i, μ_i is its centroid, k is the number of clusters, and the norm is usually Euclidean. The algorithm alternates between assigning each feature vector to the nearest centroid and recomputing centroids. It converges to a local minimum, so initialization can affect the result.

Feature scaling is essential. If intensity is measured on a 0–255 scale and spatial coordinates on a 0–1000 scale, position may dominate unless features are normalized or weighted. The number of clusters must be chosen from prior knowledge, validation, or criteria such as silhouette analysis. K-means assumes compact, approximately spherical clusters and is not inherently aware of object connectivity.

Fuzzy c-means allows partial membership. This is useful at boundaries, but the resulting memberships still require a decision rule. Supervised classifiers can incorporate labelled examples, although their reliability depends on representative training data and control of domain shift.

6. Watershed Segmentation

The watershed transform interprets a grayscale image as a topographic surface. Local minima form catchment basins; flooding from minima produces watershed lines. Applied directly to a gradient image, it tends to place boundaries near strong edges. However, noise creates many local minima and causes severe over-segmentation.

Marker-controlled watershed addresses this problem. Foreground markers identify likely object interiors, background markers identify non-object areas, and the transform is constrained to produce regions associated with these markers. Markers can be generated by thresholding, distance transforms, or connected components. For touching objects, peaks in a distance transform often provide separate foreground markers.

The distance transform assigns each foreground pixel its distance to the nearest background pixel. For approximately circular touching objects, local maxima correspond roughly to object centres. The quality of watershed segmentation depends more on marker construction than on the flooding operation itself.

7. Graph-Based Segmentation

Graph methods represent pixels or superpixels as vertices connected by weighted edges. A weight may measure colour similarity, spatial proximity, or boundary strength. Segmentation seeks a partition that balances internal similarity and separation between regions.

In a normalized-cut formulation, a graph is divided by minimizing a cost related to connections across the partition relative to total connections. In energy-based binary segmentation, labels are selected by minimizing

Here L_p is the label at pixel p, D_p is a data cost expressing compatibility with that label, V_p,q is a pairwise smoothness cost for neighbouring pixels, 𝒩 is the neighbourhood edge set, and λ controls the trade-off between data fidelity and smoothness. Units are determined by the chosen cost definitions; λ must therefore be tuned consistently.

Graph cuts can incorporate colour models, user-provided seeds, and spatial regularization. They are powerful but require greater computational and modelling effort than thresholding. Excessive smoothness can erase thin structures or merge adjacent objects.

8. Background Modelling for Video

When a camera is fixed and the background is relatively stable, moving objects can be detected by comparing the current frame with a background estimate B_t(x,y). Simple frame differencing uses

and labels foreground when D_t(x,y) > τ. A running-average background model may be updated as

where 0 < α ≤ 1 is the learning rate. A large α adapts quickly but may absorb slowly moving objects into the background; a small value is stable but adapts slowly to illumination changes.

Real scenes often require adaptive statistical models, such as mixtures of Gaussians, to represent periodic motion, waving foliage, or multiple background states. Shadows may be reduced by colour and chromaticity tests. Camera vibration, automatic exposure, and background changes remain important failure sources.

9. Mathematical Morphology

Mathematical morphology analyses shapes using a set or function called a structuring element. For a binary image, foreground is represented as a set A of pixel coordinates, and the structuring element is B. Its origin and shape must be specified.

9.1 Erosion and dilation

Binary erosion is

where B_z is B translated by z. Erosion removes foreground boundary pixels and eliminates structures smaller than the structuring element.

Binary dilation is

where B̂ is the reflection of B about its origin. Dilation expands foreground regions, fills small gaps, and connects nearby components. For symmetric structuring elements, reflection does not change the element.

For grayscale morphology, erosion and dilation are commonly defined using local minimum and maximum operations. For grayscale image f and structuring function b, one standard flat-element form is

Boundary handling must be defined, for example by padding or restricting the valid domain.

9.2 Opening and closing

Opening is erosion followed by dilation:

It removes small foreground objects and smooths narrow protrusions while generally preserving larger object structure. Closing is dilation followed by erosion:

It fills small holes and connects narrow breaks. Opening is not simply “noise removal” in all circumstances, and closing can merge distinct objects; the element size must reflect physical dimensions.

9.3 Morphological reconstruction and shape operations

Morphological reconstruction uses a marker image constrained by a mask. It is useful for removing border-connected objects, extracting regional maxima, and preserving the contours of significant objects better than ordinary opening. Connected-component labelling then assigns an identifier to each connected foreground region. With 4-connectivity, diagonal pixels are not connected; with 8-connectivity, they may be connected. This choice affects object counts and measurements.

The morphological gradient, often defined as dilation minus erosion, emphasizes boundaries:

It is distinct from the differential gradient and depends on the structuring element.

10. Choosing Structuring Elements

A disk is suitable for approximately isotropic objects, a line is useful for directional defects or elongated features, and a rectangle matches grid-like structures. The element size should be related to image resolution and the smallest feature to preserve. If a scratch is two pixels wide and a five-pixel disk is used for opening, the scratch may disappear. Conversely, a one-pixel element may have little practical effect.

Morphology is not a substitute for correct acquisition. Excessive operations can distort areas, perimeter, centroids, and defect dimensions. Quantitative inspection should measure objects after a validated sequence and should report the processing parameters.

11. Robustness Analysis

Illumination

Uneven lighting undermines global thresholding and colour rules. Diffuse lighting, flat-field correction, adaptive thresholds, background normalization, and photometric calibration improve robustness. Normalization must not remove the physical contrast required to detect the object.

Texture

Texture produces many edges and can violate region homogeneity. Multi-scale smoothing, texture descriptors, superpixels, or learned classifiers may help. A method should distinguish object boundaries from internal texture rather than assuming every edge is a region boundary.

Noise

Sensor noise creates false edges and local extrema. Gaussian or median filtering may improve segmentation, but smoothing reduces spatial detail. The filter should be selected according to noise type: median filtering is often effective for impulse noise, while Gaussian filtering is analytically convenient for approximately Gaussian noise.

Occlusion and touching objects

Occlusion removes evidence and makes complete object recovery impossible from one view without prior assumptions. Multiple cameras, temporal tracking, shape models, depth sensing, or re-identification can help. Touching objects may require distance-transform markers and watershed, contour concavity analysis, or a learned instance-segmentation model.

Evaluation

For a binary mask, true positives, false positives, true negatives, and false negatives are counted relative to a reference annotation. Important measures include

and intersection over union:

where P is the predicted foreground set and G is the ground-truth set. Precision penalizes false detections, recall penalizes missed foreground, and IoU measures overlap. Boundary accuracy, object-count accuracy, processing time, and downstream dimensional error may be more relevant than pixel accuracy in an industrial system.

12. Applied Case: Illustrative Conveyor Inspection

Background

Consider an illustrative automated inspection cell for aluminium housings moving on a conveyor. A fixed camera views the top surface. The objective is to locate each housing, reject parts with large dark dents, and provide the robot with the centroid and orientation of accepted parts.

Problem

Initial global thresholding worked during morning tests but failed later because sunlight entering the factory changed the conveyor brightness. Dark dents were sometimes merged with the housing boundary, while shadows were falsely classified as defects.

Analysis

The housing and conveyor had partially overlapping grayscale ranges under variable illumination. The part colour was relatively stable in a calibrated colour space, and the background geometry was fixed. The defect criterion depended on dark regions inside the housing, not merely on external contours. Therefore, one segmentation operation was insufficient.

Approach and solution

The camera exposure was locked and diffuse lighting was installed. A background mask was obtained from conveyor-only images and refined using morphological closing. The housing candidate was segmented using an adaptive colour-intensity rule. Small isolated regions were removed by opening, and holes caused by highlights were filled by closing. Connected components were filtered by expected area and aspect ratio.

For each accepted housing, an interior region was eroded from the boundary to avoid shadows at the edge. Dark-defect candidates were detected by adaptive thresholding within this interior. Candidate regions were retained only when their area exceeded a validated minimum and their contrast relative to a local background exceeded a specified tolerance. The centroid was computed from the binary component. If pixel coordinates are (x_i,y_i) for N foreground pixels, the centroid is

The orientation was estimated from the principal axis of the component or from a fitted contour. Calibration then converted image coordinates into conveyor coordinates.

Outcome

The system became less sensitive to gradual illumination changes, and the interior-region rule reduced false defect detections near housing boundaries. Validation was performed across shifts, part orientations, clean and dusty conveyor conditions, and deliberately introduced defects. The key performance measures were missed defective parts, false rejects, centroid error, and cycle time rather than segmentation IoU alone.

Learning points

The case demonstrates layered segmentation: acquisition control, object extraction, morphology, region restriction, defect segmentation, and geometric measurement. It also shows why a single global threshold is often fragile in real production environments.

13. Implementation and Professional Practice

A practical segmentation pipeline should separate acquisition, preprocessing, segmentation, post-processing, measurement, and decision logic. Parameters should be stored with version information and tested on representative datasets. Thresholds and structuring-element sizes should not be tuned only on ideal images. The validation set should include normal variation and known failure cases.

Real-time systems must consider latency, memory, camera exposure time, and synchronization with actuators. A segmentation result that arrives after the part has passed the reject mechanism is operationally incorrect. Safety-critical decisions require monitoring, fail-safe behaviour, traceability, and human review where appropriate.

AI-based segmentation can improve performance for complex texture, occlusion, and variable appearance. However, a learned model requires labelled data, performance monitoring, protection against domain shift, and explanation of uncertainty. Classical morphology remains valuable for post-processing neural masks, enforcing geometric constraints, and implementing auditable rules. Privacy, data governance, and secure storage are relevant when images contain people, production information, or proprietary designs.

Chapter Summary

Segmentation converts pixels into meaningful regions for measurement and control. Global thresholding is fast but assumes stable class separation; adaptive and colour-based methods address spatial or chromatic variation. Edge methods detect intensity transitions, while region growing and split-and-merge exploit local homogeneity. Clustering organizes feature vectors, watershed separates topographic basins and touching objects when markers are reliable, and graph-based methods optimize data and smoothness costs. Background modelling is effective for fixed-camera video but must handle illumination and scene changes.

Mathematical morphology uses structuring elements to manipulate shape. Erosion removes, dilation expands, opening suppresses small foreground structures, and closing fills gaps and holes. Reconstruction and connected components support more selective processing and measurement. Robust design requires attention to lighting, texture, noise, occlusion, connectivity, parameter scale, computational timing, and evaluation metrics. In professional systems, segmentation should be validated as part of the complete sensing-to-action pipeline.

Key Terms

Segmentation: Partitioning an image into meaningful regions or labels.

Binary mask: An image assigning foreground and background labels.

Threshold: A decision value used to classify image measurements.

Otsu’s method: Automatic threshold selection by maximizing between-class variance.

Adaptive thresholding: Thresholding using local image statistics.

Gradient: Vector of spatial intensity derivatives.

Edge: A location of significant image change.

Region growing: Segmentation by adding similar neighbouring pixels from seeds.

Clustering: Grouping feature vectors according to similarity.

Watershed: Topographic segmentation based on catchment basins.

Marker-controlled watershed: Watershed constrained by specified seed regions.

Graph-based segmentation: Segmentation using vertices, edges, and an optimization cost.

Background model: An estimate of scene appearance without moving foreground objects.

Structuring element: Shape used to probe an image in morphology.

Erosion: Morphological operation that removes boundary pixels.

Dilation: Morphological operation that expands foreground regions.

Opening: Erosion followed by dilation.

Closing: Dilation followed by erosion.

Connected component: A maximally connected foreground region.

Intersection over union: Overlap measure between predicted and reference regions.

Review Questions

Short Answer Questions

1. What assumption is required for global thresholding to work reliably?
2. Distinguish erosion from dilation.
3. Why can direct watershed segmentation over-segment an image?
4. What is the role of the learning rate in a running-average background model?
5. Define intersection over union.

Long Answer Questions

1. Compare global thresholding, adaptive thresholding, and colour-based segmentation, including their assumptions and failure modes.
2. Explain the Canny edge-detection pipeline and the effect of Gaussian scale.
3. Describe region growing, split-and-merge, and clustering as alternative region-based strategies.
4. Explain opening, closing, reconstruction, and connected-component analysis in an industrial inspection pipeline.
5. Discuss how illumination, texture, noise, and occlusion influence segmentation design.

Analytical / Application Questions

1. A fixed-camera conveyor system experiences slowly changing illumination and intermittent shadows. Design a segmentation pipeline and justify each stage.
2. Two circular parts touch in a binary image. Explain how a distance transform and marker-controlled watershed could separate them, and identify likely failure modes.
3. A defect detector has TP = 90, FP = 10, and FN = 20. Calculate precision, recall, and IoU, and explain which operational error each metric emphasizes.

Multiple Choice Questions with Answers

Question 1

Which condition most strongly supports the use of global thresholding?

A. A stable intensity separation between foreground and background

B. Severe spatially varying illumination

C. Objects with no measurable contrast

D. A camera undergoing continuous vibration

Correct Answer: A. A stable intensity separation between foreground and background

Question 2

In Otsu’s method, the selected threshold maximizes which quantity?

A. Image perimeter

B. Between-class variance

C. Number of connected components

D. Gaussian filter width

Correct Answer: B. Between-class variance

Question 3

What is the usual effect of binary erosion on a foreground region?

A. It expands the region

B. It removes boundary pixels

C. It changes every pixel to its local average

D. It assigns random labels

Correct Answer: B. It removes boundary pixels

Question 4

Which morphological sequence is an opening?

A. Dilation followed by erosion

B. Erosion followed by dilation

C. Thresholding followed by clustering

D. Gradient followed by convolution

Correct Answer: B. Erosion followed by dilation

Question 5

Why are markers used in marker-controlled watershed segmentation?

A. To increase camera resolution

B. To constrain basin formation and reduce over-segmentation

C. To remove all object boundaries

D. To convert grayscale images into RGB images

Correct Answer: B. To constrain basin formation and reduce over-segmentation

Question 6

In a running-average background model, increasing α generally causes the model to:

A. Adapt more quickly to recent frames

B. Become completely independent of recent frames

C. Eliminate all foreground objects immediately

D. Reduce the image dimensions

Correct Answer: A. Adapt more quickly to recent frames

Question 7

Given TP = 80 and FP = 20, what is precision?

A. 0.20

B. 0. cinquante

C. 0.80

D. 0.833

Correct Answer: D. 0.833

Question 8

Which feature is most directly emphasized by a gradient-based edge detector?

A. Rapid spatial intensity change

B. Object class name

C. Camera frame rate

D. Connected-component area only

Correct Answer: A. Rapid spatial intensity change

Question 9

Which structuring element is generally most appropriate for detecting or processing elongated defects in a known direction?

A. A directional line element

B. A random pixel set

C. A full-image rectangle only

D. A colour histogram

Correct Answer: A. A directional line element

Question 10

In the graph-based energy E(L) = data + λ smoothness, increasing λ generally gives greater importance to:

A. Spatial smoothness between neighbouring labels

B. Camera exposure time

C. Pixel bit depth

D. The number of image channels

Correct Answer: A. Spatial smoothness between neighbouring labels