10. Feature Extraction, Representation and Object Analysis

Introduction and Context

Image segmentation separates an image into meaningful regions, but segmentation alone does not identify what those regions represent. A binary object mask, for example, indicates which pixels belong to an object but says little about its shape, orientation, texture, or identity. Feature extraction converts image data into measurements that describe visually significant properties. Representation organizes these measurements so that objects can be compared, classified, detected, tracked, or recognized.

Feature extraction is therefore an intermediate layer between low-level image formation and high-level computer vision. Its design strongly influences the reliability of an entire system. A face-recognition system, an industrial inspection system, and a medical-image analysis system may all use the same image-processing operations, yet require very different representations. The appropriate feature is not simply the one that is easiest to calculate; it is the one that preserves information relevant to the task while reducing irrelevant variation caused by illumination, noise, scale, viewpoint, or sensor characteristics.

This chapter examines boundaries, regions, texture, shape, moments, corners, local descriptors, dimensionality reduction, and invariant representations. It also considers how these components are integrated into practical object-analysis pipelines and how classical descriptors relate to contemporary learned representations.

Learning Objectives

After studying this chapter, the reader should be able to:

1. distinguish boundary-based, region-based, texture-based, and local features;

2. compute and interpret geometric descriptors, moments, and shape measures;

3. explain corner detection and local feature-description methods;

4. select representations according to invariance, discriminability, and computational requirements;

5. apply dimensionality-reduction methods such as PCA appropriately;

6. design an object-analysis workflow for detection, tracking, or recognition;

7. evaluate practical limitations, failure modes, and ethical implications of feature-based systems.

From Pixels to Descriptors

A feature is a measurable property derived from an image. Let an image be represented by an intensity function \(I(x,y)\), and let a segmented object be represented by a region \(R\) or its boundary \(\partial R\). A feature extractor is a mapping

\[

\phi(I,R) = [f_1,f_2,\ldots,f_d]^T,

\]

where the resulting vector is called a feature vector or descriptor. The dimensionality \(d\) may be small, as in area and perimeter, or large, as in a histogram of local gradient descriptors.

A useful representation should have four properties. It should be discriminative, meaning that different object classes produce distinguishable descriptors. It should be stable, so small image noise does not cause large descriptor changes. It should possess appropriate invariance to transformations such as translation, rotation, or scale. Finally, it should be computationally practical for the intended application.

These properties often conflict. A highly detailed descriptor may be discriminative but sensitive to viewpoint and expensive to compute. A compact descriptor may be efficient but unable to distinguish similar objects. Feature design is consequently a problem of selecting and balancing information.

Boundary and Region Descriptors

Boundary representation

A boundary is the set of pixels separating an object from its background. It may be represented as an ordered contour, a chain code, a polygonal approximation, or a sampled radial function. Boundary representations are especially useful when the outline carries most of the object identity, as in character recognition, industrial silhouettes, and biological shape analysis.

A common boundary measure is perimeter. In a digital image, perimeter can be estimated by counting horizontal, vertical, and diagonal boundary transitions, although the estimate depends on pixel connectivity and the chosen correction scheme. Boundary curvature describes how rapidly the contour direction changes. Sharp curvature points may identify meaningful parts, but curvature is sensitive to noise; smoothing is usually required before differentiation.

Chain codes encode successive directions between boundary pixels. A four-direction code uses horizontal and vertical moves, whereas an eight-direction code also includes diagonals. Chain codes are compact, but their raw numerical values depend on the starting point and orientation. Differential chain codes, which encode changes between successive directions, reduce some of this dependence.

Polygonal approximation replaces a detailed contour with a smaller sequence of line segments. The Douglas–Peucker method recursively removes points whose distance from an approximating segment is below a tolerance. A small tolerance preserves detail but retains noise; a large tolerance produces a compact representation but may remove important shape structure.

Region representation

Region descriptors use all pixels inside an object rather than only its boundary. The area is the number of object pixels, while the centroid is

\[

\bar{x}=\frac{1}{A}\sum_{(x,y)\in R}x, \qquad

\bar{y}=\frac{1}{A}\sum_{(x,y)\in R}y.

\]

The bounding box, extent, filled area, and convex-hull area provide additional geometric information. Solidity, defined as object area divided by convex-hull area, is useful for identifying concavity. Extent, defined as object area divided by bounding-box area, indicates how fully the object occupies its box.

Compactness is commonly expressed as

\[

C=\frac{4\pi A}{P^2},

\]

where \(A\) is area and \(P\) is perimeter. A circle has the maximum ideal value of one; elongated or irregular objects have lower values. In practice, digitization and segmentation noise affect the estimate.

The eccentricity of an ellipse fitted to the object indicates elongation. The major and minor axes can be obtained from the eigenvectors and eigenvalues of the covariance matrix of object coordinates. These descriptors are simple, interpretable, and efficient, but they may be inadequate when objects have similar global geometry or when appearance and internal structure are important.

Moments and Shape Structure

Image moments provide a mathematical framework for describing mass distribution. The raw geometric moment of order \(p+q\) is

\[

m_{pq}=\sum_x\sum_y x^p y^q I(x,y).

\]

For a binary object, \(I(x,y)\) is usually one inside the region and zero elsewhere. Thus \(m_{00}\) equals area, and the first-order moments determine the centroid:

\[

\bar{x}=m_{10}/m_{00}, \qquad \bar{y}=m_{01}/m_{00}.

\]

Central moments remove translation dependence:

\[

\mu_{pq}=\sum_x\sum_y (x-\bar{x})^p(y-\bar{y})^q I(x,y).

\]

Normalized central moments reduce dependence on scale. From these quantities, Hu proposed seven moment invariants that are theoretically invariant to translation, scaling, and rotation. They are widely used as compact shape descriptors, although their numerical values can span several orders of magnitude and are sensitive to segmentation errors. Logarithmic transformations are often applied before classification.

Moments also support orientation estimation. The second-order central-moment matrix, or covariance matrix, describes the distribution of pixels around the centroid. Its principal eigenvector gives the dominant object axis. This orientation is useful for aligning objects before comparison, but it becomes unstable for nearly circular or symmetric objects.

Texture as an Object Attribute

Shape may not distinguish objects with similar outlines. Texture describes spatial variation in intensity, color, or local structure. It can be analyzed statistically, structurally, or through multiscale frequency methods.

The gray-level co-occurrence matrix (GLCM) records how often intensity pair \((i,j)\) occurs at a specified displacement and direction. Important measures include contrast, energy, homogeneity, correlation, and entropy. GLCM features depend on quantization level, displacement, and direction; averaging across directions can improve rotation tolerance but may discard directional information.

The local binary pattern (LBP) operator compares a central pixel with neighboring pixels. Each neighbor contributes a binary value according to whether it is greater than or equal to the center. The resulting binary word is converted into a histogram over local regions. LBP is computationally efficient and relatively tolerant of monotonic illumination changes, but it is sensitive to noise and may lose information about absolute intensity and large-scale structure.

Filter-bank methods, including Gabor filters, analyze texture at multiple orientations and spatial frequencies. A Gabor response is strong when local image structure matches the filter’s frequency and direction. Such features are useful for fabric inspection, fingerprint analysis, and material classification, though they may require many filters and produce high-dimensional vectors.

Texture should be computed over regions large enough to capture repeated patterns but small enough to avoid mixing different materials. This trade-off is important in segmentation, medical imaging, and remote sensing.

Corners, Interest Points, and Local Descriptors

Corners are locations where image intensity changes significantly in two directions. They are valuable for registration, tracking, and matching because their neighborhoods are more distinctive than those of uniform edges.

The Harris detector analyzes the local second-moment matrix

\[

M=\sum_{(u,v)\in W} w(u,v)

\begin{bmatrix}I_x^2&I_xI_y\\I_xI_y&I_y^2\end{bmatrix},

\]

where \(I_x\) and \(I_y\) are image gradients and \(W\) is a neighborhood. Its response is commonly written as

\[

R=\det(M)-k(\operatorname{trace}M)^2.

\]

Large positive responses indicate strong variation in two directions. Non-maximum suppression selects spatially distinct interest points. The Shi–Tomasi method uses the smaller eigenvalue of \(M\), selecting points that are strong in both directions.

A detector identifies a point; a descriptor describes its neighborhood. The scale-invariant feature transform (SIFT) detects extrema in a scale space and describes each point using a histogram of local gradient orientations. The descriptor is designed to be relatively robust to scale and rotation and moderately tolerant of illumination and viewpoint changes.

Speeded-Up Robust Features (SURF) approximate scale-space operations using box filters and integral images. ORB combines an oriented FAST detector with a rotated BRIEF binary descriptor. Binary descriptors are fast to match using Hamming distance and are attractive in embedded and real-time systems, although their robustness may differ from that of gradient-histogram descriptors under large viewpoint or illumination changes.

A typical matching pipeline detects keypoints, computes descriptors, compares descriptors using an appropriate distance, rejects ambiguous matches, and estimates a geometric transformation using a robust method such as RANSAC. RANSAC is important because some descriptor matches are inevitably incorrect.

Invariance and Canonical Representation

An object may appear at different image locations, sizes, orientations, illuminations, and viewpoints. A representation is translation invariant if shifting the object does not change its descriptor; scale invariant if resizing has little effect; and rotation invariant if image rotation does not alter the representation. Illumination invariance means that brightness or contrast changes have limited influence.

Invariance can be achieved by normalization, canonical alignment, or descriptor construction. Subtracting the centroid addresses translation. Dividing coordinates by a characteristic size addresses scale. Aligning the principal axis addresses rotation, although symmetry can make the axis ambiguous. Gradient normalization and local contrast normalization reduce illumination effects.

Invariance should not be maximized blindly. If orientation is meaningful—for example, in reading text or identifying vehicle direction—removing orientation may reduce discriminative power. The correct design depends on which transformations represent nuisance variation and which carry class information.

Dimensionality Reduction and Feature Selection

Feature vectors may contain redundant, correlated, or noisy components. High dimensionality increases storage and computational cost and can impair statistical learning, especially when the number of training examples is limited.

Principal component analysis (PCA) transforms centered data into orthogonal directions of decreasing variance. If the covariance matrix is \(S\), its eigenvectors define principal directions. Retaining the first \(k\) components gives a lower-dimensional approximation that minimizes reconstruction error among linear subspaces of dimension \(k\). PCA is unsupervised: high variance does not necessarily mean high class discrimination.

Linear discriminant analysis (LDA), in contrast, seeks projections with high between-class variation and low within-class variation. It is supervised and can be effective for classification, but it relies on assumptions about class distributions and has a dimensionality limit related to the number of classes.

Feature selection retains existing variables rather than constructing new combinations. Filters use statistical criteria, wrappers evaluate subsets through a classifier, and embedded methods perform selection during model training. Standardization is essential when features have different units or scales. Dimensionality reduction must be fitted on training data only; applying information from the test set creates data leakage.

An Integrated Object-Analysis Workflow

A robust object-analysis system generally follows these stages:

1. acquire and calibrate the image;

2. reduce noise while preserving relevant boundaries;

3. segment candidate objects;

4. remove artifacts and label connected components;

5. compute complementary features, such as geometry, moments, texture, and local descriptors;

6. normalize or align descriptors where appropriate;

7. reduce dimensions or select features using training data;

8. classify, match, detect, or track objects;

9. validate results using task-specific metrics.

The stages are not strictly independent. A poor segmentation produces unreliable moments and texture measurements. Conversely, a strong descriptor may tolerate imperfect segmentation. Feature fusion may be early, by concatenating vectors, or late, by combining decisions from separate classifiers. Early fusion can exploit relationships among features but may create scaling and dimensionality problems. Late fusion is more modular and may be easier to interpret.

For detection, precision, recall, intersection over union, and average precision are common measures. For tracking, identity switches, localization accuracy, and track continuity matter. For recognition, confusion matrices, class-wise recall, and calibration should be examined rather than relying only on overall accuracy.

Applications in Practice

In industrial inspection, region area, perimeter, convexity, texture, and local defects can identify scratches, missing components, or incorrect assembly. In medical imaging, shape and texture descriptors may characterize lesions, cells, or anatomical structures, but the descriptors should support clinical judgment rather than replace it without validation. In remote sensing, spectral, texture, and shape features help distinguish land-cover classes. In document analysis, connected components, projection profiles, contours, and local descriptors support character and layout recognition.

In video analytics, local features support camera motion estimation and object matching, while region descriptors support tracking. A tracker may predict an object location using a motion model and then update it using appearance features. If appearance is represented only by color, objects with similar colors may be confused; combining color, shape, and local texture can improve identity maintenance.

Illustrative Case Study: Automated Inspection of Metal Components

Background

Consider a fictional manufacturing line producing circular metal washers. A camera captures each washer on a conveyor. The objective is to reject components with an incorrect outer diameter, an oversized central hole, cracks, or severe surface defects.

Situation and problem

Initial inspection based only on thresholded area produced false decisions. Variations in illumination changed the apparent boundary, and washers with the correct outline but surface scratches were accepted.

Analysis

The task contains two distinct information sources. Outer and inner contours determine geometry, whereas scratches and pitting are texture or local-contrast phenomena. A single feature cannot reliably represent both.

Approach

The system first normalizes illumination and segments the metal from the conveyor background. Morphological opening removes isolated noise, and connected-component analysis isolates each washer. Contours provide outer and inner perimeters, while moments estimate centroids and orientation. The ratio of inner to outer area, circularity, solidity, and radial boundary deviation describe geometry. A local texture descriptor is computed within the annular region, excluding the hole and a narrow boundary band. PCA is fitted on the training set to compress the combined descriptor. A classifier then distinguishes acceptable and defective components. RANSAC-based alignment is used when the camera position changes slightly between batches.

Outcome

The combined representation separates geometric defects from surface defects more effectively than area alone. Validation is performed by defect type, since a high overall accuracy could conceal poor detection of rare cracks. Thresholds are reviewed with production engineers because false acceptance and false rejection have different costs.

Learning points

The case illustrates four principles: descriptors should reflect the physical source of variation; segmentation quality affects every downstream measurement; feature normalization is essential when combining unlike measurements; and evaluation must consider operational costs, not only average accuracy.

Limitations, Risks, and Common Mistakes

Feature extraction is not automatically robust. Boundary descriptors fail when segmentation is incomplete or when occlusion removes distinctive parts. Region descriptors may confuse objects with similar mass distribution. Texture measures are affected by resolution, illumination, and window size. Local descriptors may produce many false matches in repetitive patterns.

Common mistakes include computing texture across multiple materials, using an unnormalized concatenated feature vector, estimating rotation from a symmetric object, fitting PCA before separating training and test data, and evaluating only on images resembling the development set. Another mistake is assuming that invariance to every transformation is desirable.

Class imbalance can make accuracy misleading. Dataset bias may cause a descriptor to work well under one camera, demographic group, or environment and fail elsewhere. Reproducibility requires recording image resolution, preprocessing, segmentation parameters, descriptor settings, training splits, and evaluation protocols.

Professional and Ethical Considerations

When object analysis is used in safety-critical, medical, employment, or surveillance contexts, performance must be measured across relevant populations and operating conditions. Decisions should be auditable, and human review should remain available where errors have serious consequences. Images may contain personal or commercially sensitive information; collection, storage, and sharing should follow applicable privacy and data-governance requirements. A technically accurate system can still be professionally unacceptable if its deployment context encourages unjustified automation or lacks mechanisms for appeal and correction.

Current Developments

Modern vision systems increasingly learn representations directly from data using convolutional networks, vision transformers, and self-supervised learning. Deep embeddings can outperform hand-crafted descriptors when large, diverse training data are available. Nevertheless, classical features remain valuable for low-power devices, small datasets, interpretability, metrology, and systems requiring explicit geometric reasoning.

Hybrid systems combine segmentation, geometric measurements, local descriptors, and learned embeddings. Feature stores, automated model monitoring, and edge inference support deployment at scale. Emerging research emphasizes domain adaptation, uncertainty estimation, equivariant representations, multimodal features, and self-supervised learning that reduces dependence on manually labeled images. These developments do not eliminate the need for feature analysis; they shift attention toward representation quality, data coverage, robustness, and responsible evaluation.

Best-Practice Guidelines

A practical feature-extraction design should:

• begin with the imaging conditions and the decision to be made;
• use complementary descriptors rather than an unnecessarily large collection;
• normalize scale, orientation, and units only when the task warrants it;
• preserve a clear separation between training, validation, and test data;
• inspect feature distributions and failure cases visually;
• report class-wise and operationally meaningful metrics;
• test robustness to noise, illumination, blur, occlusion, and viewpoint;
• retain interpretable measurements when they support engineering or clinical review;
• document parameters and version the complete preprocessing pipeline.

Chapter Summary

Feature extraction transforms pixels, boundaries, regions, and local neighborhoods into representations suitable for object analysis. Boundary descriptors capture contour structure; region descriptors summarize area, centroid, compactness, convexity, and elongation; moments provide translation- and scale-normalized shape information; and texture descriptors characterize spatial intensity variation. Corners and local descriptors support matching, registration, detection, and tracking. Invariance must be selected according to the task rather than applied indiscriminately. PCA, LDA, and feature-selection methods reduce redundancy, but they must be used without data leakage. Reliable systems integrate segmentation, descriptor computation, normalization, dimensionality reduction, decision-making, and task-specific evaluation. Classical descriptors remain useful alongside learned representations, particularly when interpretability, limited data, or computational efficiency matters.

Key Terms

Feature vector: A numerical vector representing measurable image properties.

Boundary descriptor: A feature derived primarily from an object’s contour.

Region descriptor: A feature computed from pixels within an object region.

Centroid: The intensity- or area-weighted average position of an object.

Compactness: A measure relating area to squared perimeter.

Solidity: Object area divided by the area of its convex hull.

Image moment: A weighted sum of image intensities and spatial coordinates.

Central moment: A moment computed relative to the object centroid.

Hu invariant: A moment combination designed to resist translation, scale, and rotation changes.

Texture: Spatial variation or organization of intensity, color, or local structure.

GLCM: A matrix describing gray-level pair frequencies at a displacement and direction.

LBP: A local texture code obtained by thresholding neighbors against a central pixel.

Corner: A location with strong intensity variation in two directions.

Interest point: A locally distinctive image location selected for analysis or matching.

Local descriptor: A numerical description of an image neighborhood around an interest point.

Invariance: Insensitivity of a representation to specified image transformations.

PCA: An unsupervised linear transformation retaining directions of greatest variance.

LDA: A supervised projection emphasizing between-class separation relative to within-class variation.

RANSAC: A robust estimation method that fits a model while rejecting outliers.

Data leakage: Use of information from evaluation data during model development.

Review Questions

Short Answer Questions

1. Differentiate a boundary descriptor from a region descriptor.

2. What information is provided by the covariance matrix of object coordinates?

3. Why are central moments less sensitive to translation than raw moments?

4. State one strength and one limitation of LBP.

5. Why is non-maximum suppression used after corner detection?

Long Answer Questions

1. Explain the computation and interpretation of area, perimeter, compactness, solidity, extent, and eccentricity.

2. Describe GLCM, LBP, and Gabor-based texture representations and compare their practical uses.

3. Explain how SIFT- or ORB-type local-feature matching can be used for object recognition.

4. Discuss translation, scale, rotation, and illumination invariance, including situations in which invariance may be undesirable.

5. Design a complete feature-extraction workflow for an industrial inspection system and justify each stage.

Analytical / Application Questions

1. Two objects have nearly identical silhouettes but different surface patterns. Which descriptors would you combine, and how would you prevent one feature family from dominating the classifier?

2. A recognition system performs well on development images but fails after camera replacement. Identify likely causes and propose an evaluation and redesign strategy.

3. A nearly circular object produces unstable orientation estimates. Explain why this occurs and recommend an alternative alignment or representation strategy.

22. Multiple Choice Questions with Answers

Question 1

Which descriptor is most directly affected by the choice of convex hull?

A. Solidity

B. LBP uniformity

C. Harris response

D. GLCM correlation

Correct Answer: A. Solidity

Question 2

What is the principal purpose of central moments?

A. To encode color channels

B. To reduce dependence on object translation

C. To detect image edges only

D. To increase image resolution

Correct Answer: B. To reduce dependence on object translation

Question 3

A circular object is enlarged uniformly. Which representation is designed to reduce the resulting change in descriptor values?

A. Scale-normalized moments

B. Raw perimeter only

C. Unnormalized pixel coordinates

D. A fixed-size intensity crop without resizing

Correct Answer: A. Scale-normalized moments

Question 4

In the Harris detector, a strong corner generally has:

A. Large intensity variation in only one direction

B. Small gradients in both directions

C. Strong variation in two independent directions

D. Uniform intensity throughout its neighborhood

Correct Answer: C. Strong variation in two independent directions

Question 5

Which method is supervised and seeks class separation rather than merely maximum variance?

A. PCA

B. LDA

C. Douglas–Peucker approximation

D. Chain coding

Correct Answer: B. LDA

Question 6

Why is RANSAC useful in local-feature matching?

A. It increases image brightness

B. It rejects geometrically inconsistent matches

C. It converts grayscale images to color

D. It computes object area directly

Correct Answer: B. It rejects geometrically inconsistent matches

Question 7

A system combines area, perimeter, and texture energy without normalization. What is the principal risk?

A. The features cannot be stored digitally

B. A feature with a larger numerical scale may dominate

C. The image will lose all boundaries

D. PCA becomes supervised automatically

Correct Answer: B. A feature with a larger numerical scale may dominate

Question 8

Which statement best describes PCA?

A. It always maximizes classification accuracy

B. It creates orthogonal directions ordered by variance

C. It detects corners using image gradients

D. It guarantees rotation invariance

Correct Answer: B. It creates orthogonal directions ordered by variance

Question 9

In the washer-inspection case, why was texture measured inside an annular region rather than across the entire object image?

A. To exclude irrelevant background and the central hole

B. To eliminate the need for segmentation

C. To make perimeter equal to area

D. To guarantee perfect illumination invariance

Correct Answer: A. To exclude irrelevant background and the central hole

Question 10

Why can indiscriminate invariance reduce recognition performance?

A. It may remove a transformation that carries useful class information

B. It always increases feature dimensionality

C. It prevents all forms of normalization

D. It makes segmentation impossible

Correct Answer: A. It may remove a transformation that carries useful class information