Object Detection, Segmentation and Keypoints

Introduction and Context

Image classification assigns one label to an entire image, but many computer-vision applications require a richer description: what objects are present, where they are located, which pixels belong to each object, and how people or articulated objects are posed. Object detection, segmentation, and keypoint estimation address these requirements.

These capabilities support autonomous driving, medical imaging, retail analytics, robotics, sports analysis, industrial inspection, augmented reality, and assistive technologies. They also introduce practical concerns absent from simple classification, including localization accuracy, overlapping instances, small objects, annotation cost, inference latency, privacy, and safety.

This chapter compares one-stage and two-stage detectors and explains the principles, losses, metrics, implementation choices, and limitations of detection, semantic segmentation, instance segmentation, and keypoint-based pose analysis.

Learning Objectives

By the end of this chapter, students should be able to:

1. Distinguish classification, detection, semantic segmentation, instance segmentation, and keypoint estimation.

2. Explain the architectures and trade-offs of one-stage and two-stage object detectors.

3. Formulate common classification, localization, overlap, segmentation, and keypoint losses.

4. Interpret detection, segmentation, and pose metrics.

5. Design a practical training and evaluation workflow.

6. Analyze an applied computer-vision system, including accuracy, latency, fairness, privacy, and safety.

1. The Prediction Tasks

1.1 Object detection

An object detector predicts a set of objects. Each prediction normally contains a class label, confidence score, and bounding box. In the common axis-aligned representation, a box is written as

\[

b=(x_{min},y_{min},x_{max},y_{max}),

\]

where coordinates refer to image pixels or normalized image dimensions. The box must satisfy \(x_{min}<x_{max}\) and \(y_{min}<y_{max}\).

A detector must solve two problems simultaneously: classification and localization. It must also decide how many objects exist. Consequently, detection is usually more difficult than image classification, especially when objects are small, partially occluded, densely packed, or visually similar.

1.2 Semantic segmentation

Semantic segmentation assigns a class to every pixel. All pixels belonging to the same class receive the same label, even when they belong to different physical objects. For example, every car may be labelled “car,” without distinguishing car 1 from car 2.

A model commonly produces logits \(z_{p,c}\) for pixel \(p\) and class \(c\). Softmax converts logits into class probabilities:

\[

p_{p,c}=\frac{e^{z_{p,c}}}{\sum_{k=1}^{C}e^{z_{p,k}}}.

\]

Here, \(C\) is the number of classes. The semantic prediction is the class with the greatest probability at each pixel.

1.3 Instance segmentation

Instance segmentation identifies both the class and the individual object. Two people standing together therefore receive separate masks. Instance segmentation is useful for counting, measuring, tracking, robotic manipulation, and autonomous navigation.

A common architecture first detects object boxes and then predicts a binary mask for each detected region. This separates the problem into object-level localization and pixel-level shape prediction.

1.4 Keypoint estimation and pose analysis

Keypoint estimation predicts meaningful anatomical or structural points, such as shoulders, elbows, wrists, hips, knees, and ankles. A pose is represented by coordinates and, often, confidence values. Pose analysis may then infer actions, joint angles, posture, or human–object interaction.

A keypoint can be represented as \((x_k,y_k,v_k)\), where \(v_k\) indicates visibility or annotation status. In heatmap-based systems, the network predicts one heatmap per keypoint; a local maximum indicates the estimated location.

2. Object-Detection Architectures

2.1 The general detection pipeline

A typical detector performs the following operations:

1. Resize and normalize the image.

2. Extract features using a backbone such as a convolutional network or vision transformer.

3. Combine information at several resolutions through a feature pyramid or related mechanism.

4. Predict class scores and localization parameters.

5. Remove duplicate predictions using post-processing.

6. Return boxes, classes, and confidence scores.

Multi-scale features are important because large objects are represented effectively at coarse resolutions, whereas small objects require relatively fine spatial grids.

2.2 Two-stage detectors

Two-stage detectors first generate candidate regions and then classify and refine them. A representative design is a region proposal network followed by region-wise feature extraction and prediction heads. The first stage proposes likely object regions; the second stage predicts class, refined box coordinates, and, in some systems, a mask.

Advantages include strong localization and often high accuracy, particularly for crowded scenes and small objects. Region-wise processing allows the second stage to focus on candidate objects. The disadvantages are greater computational cost, more complicated training, and higher latency than many one-stage systems.

Two-stage detectors are appropriate when accuracy is more important than real-time speed, such as medical analysis, offline image annotation, and high-value inspection.

2.3 One-stage detectors

One-stage detectors predict classes and boxes directly from dense image features, without an explicit region-proposal stage. Predictions may be made at grid locations, from predefined anchors, or from anchor-free reference points. Modern one-stage detectors commonly use multi-scale feature maps and carefully designed label assignment and loss functions.

Their principal advantages are speed, simpler inference, and suitability for edge devices. Historically, they sometimes underperformed two-stage models on small or crowded objects, but architectural and training improvements have substantially reduced that gap. The main challenges remain class imbalance, duplicate predictions, difficult positive–negative assignment, and sensitivity to small-object resolution.

2.4 One-stage versus two-stage comparison

The distinction is architectural rather than an absolute ranking of quality.

| Criterion | One-stage detector | Two-stage detector |

|---|---|---|

| Region proposals | Not explicit | Explicit first stage |

| Inference speed | Usually higher | Usually lower |

| Implementation | Relatively direct | More complex |

| Small or crowded objects | Can be challenging | Often strong |

| Edge deployment | Frequently suitable | May require optimization |

| Typical use | Real-time video and robotics | High-accuracy analysis |

The correct choice depends on the error cost, available hardware, image resolution, object density, and latency requirement. Benchmark accuracy alone should not determine deployment decisions.

3. Detection Representations and Losses

3.1 Box parameterization

A detector may predict box corners directly or predict offsets relative to an anchor or reference point. For an anchor with center \((x_a,y_a)\), width \(w_a\), and height \(h_a\), a predicted center and size can be parameterized as

\[

x=x_a+t_xw_a,\quad y=y_a+t_yh_a,

\]

\[

w=w_a e^{t_w},\quad h=h_a e^{t_h}.

\]

The exponential ensures positive width and height. The parameters \(t_x,t_y,t_w,t_h\) are learned values. This parameterization is useful when anchors are used; anchor-free models use alternative distances or point-based representations.

3.2 Classification and objectness

For a binary objectness target \(y\in\{0,1\}\) and predicted probability \(p\), binary cross-entropy is

\[

L_{BCE}=-[y\log p+(1-y)\log(1-p)].

\]

It heavily penalizes confident incorrect predictions. In dense detection, negative locations can greatly outnumber positive locations, so focal loss may reduce the contribution of easy negatives:

\[

L_{focal}=-\alpha(1-p_t)^\gamma\log(p_t),

\]

where \(p_t=p\) for a positive example and \(p_t=1-p\) for a negative example. The parameters \(\alpha\) and \(\gamma\geq0\) control class weighting and focusing. Focal loss is useful under severe foreground–background imbalance but is not automatically superior in every dataset.

3.3 Localization losses

A basic coordinate loss is smooth L1 loss. For error \(r\),

\[

SmoothL1(r)=

\begin{cases}

\frac{1}{2}r^2,& |r|<1,\\

|r|-\frac{1}{2},& |r|\geq1.

\end{cases}

\]

It is quadratic near zero and linear for large errors, reducing sensitivity to outliers compared with squared error.

Overlap-based losses compare predicted and target regions directly. Intersection over Union (IoU) is

\[

IoU(B_p,B_g)=\frac{|B_p\cap B_g|}{|B_p\cup B_g|},

\]

where \(B_p\) and \(B_g\) are predicted and ground-truth boxes. IoU ranges from 0 to 1 for ordinary boxes. IoU-based losses, generalized IoU, and related distance-aware losses encourage geometrically meaningful predictions. A useful training objective is often a weighted sum:

\[

L=\lambda_{cls}L_{cls}+\lambda_{box}L_{box}+\lambda_{obj}L_{obj},

\]

where the weights must be tuned according to the model, data, and implementation. The terms have different scales, so blindly assigning equal weights can produce poor optimization.

3.4 Non-maximum suppression

Dense detectors often produce several boxes for one object. Non-maximum suppression (NMS) keeps the highest-scoring box and removes boxes whose IoU with it exceeds a selected threshold. It repeats this process for remaining boxes, usually separately by class.

NMS can accidentally remove distinct nearby objects, particularly in crowds. Soft-NMS reduces scores rather than immediately deleting overlapping boxes. The threshold is a deployment parameter, not a universal constant; it should be selected using validation data and the application’s error costs.

4. Segmentation Models and Losses

4.1 Semantic segmentation architectures

An encoder extracts increasingly abstract features while reducing spatial resolution. A decoder restores resolution and combines deep semantic information with earlier high-resolution features through skip connections. U-shaped architectures, atrous-convolution systems, and encoder–decoder transformer models are common design families.

A pixel-wise cross-entropy loss is

\[

L_{CE}=-\frac{1}{N}\sum_{p=1}^{N}\sum_{c=1}^{C} y_{p,c}\log p_{p,c},

\]

where \(N\) is the number of pixels, \(y_{p,c}\) is a one-hot target, and \(p_{p,c}\) is the predicted probability. Class weighting may be required when foreground classes occupy far fewer pixels than the background.

For binary masks, the Dice coefficient is

\[

Dice=\frac{2\sum_p y_p\hat y_p+\epsilon}{\sum_p y_p+\sum_p\hat y_p+\epsilon},

\]

where \(y_p\) and \(\hat y_p\) are target and predicted mask values, and \(\epsilon>0\) prevents division by zero. Dice loss is commonly \(1-Dice\). It is useful for imbalanced foreground–background segmentation, although its behavior can differ from pixel accuracy.

4.2 Instance segmentation

An instance-segmentation model commonly predicts a box and a mask for each detected object. The mask head may operate on a fixed-size region of interest and output a binary mask. The total loss can combine classification, box regression, objectness or proposal terms, and mask binary cross-entropy.

Mask quality depends on both detection quality and boundary precision. A correct mask cannot compensate for a missed object, and a well-localized box does not guarantee an accurate boundary. Applications requiring measurement should evaluate boundary quality and not rely solely on box AP.

5. Keypoints and Pose Estimation

5.1 Coordinate regression and heatmaps

Direct coordinate regression predicts \((x_k,y_k)\) for each keypoint. It is compact but may be difficult to optimize because a small coordinate error is not equally meaningful at all image scales.

Heatmap methods predict a spatial distribution for each keypoint. A ground-truth heatmap is often a Gaussian centered at the annotated point:

\[

H_k(x,y)=\exp\left(-\frac{(x-x_k)^2+(y-y_k)^2}{2\sigma^2}\right),

\]

where \(\sigma\) controls spread in pixels at the heatmap resolution. The model may use mean squared error between predicted and target heatmaps, or a related classification objective. Coordinates can be recovered from the maximum or a soft-argmax operation.

5.2 Visibility and uncertainty

Keypoints may be invisible because of occlusion or truncation. Training should distinguish “not visible but annotated” from “not labelled.” Visibility-aware losses prevent the model from being penalized for an unknowable location. Confidence estimates are valuable in safety-related systems; a pose should not be treated as reliable merely because the network always outputs coordinates.

5.3 Pose analysis

After keypoints are estimated, derived quantities can be computed. For three points \(A\), \(B\), and \(C\), the angle at \(B\) is

\[

\theta=\cos^{-1}\left(\frac{(A-B)\cdot(C-B)}{\|A-B\|\|C-B\|}\right),

\]

provided neither vector has zero length. The result is in radians unless converted to degrees. Image-plane angles are affected by camera viewpoint; three-dimensional pose or calibrated multi-camera data is required for reliable physical joint angles in general.

Temporal smoothing, tracking, and action models can reduce frame-to-frame jitter, but excessive smoothing can hide rapid movements. Pose systems should report uncertainty and handle missing keypoints rather than silently interpolating implausible anatomy.

6. Evaluation Metrics

6.1 Detection metrics

A detection is a true positive when its predicted class matches the ground truth and its IoU exceeds a specified threshold, subject to matching rules. A false positive is an incorrect or duplicate detection; a false negative is a missed object.

\[

Precision=\frac{TP}{TP+FP},\qquad Recall=\frac{TP}{TP+FN}.

\]

Average precision (AP) summarizes the precision–recall curve for one class. Mean average precision (mAP) averages AP across classes and, depending on the benchmark, across IoU thresholds. Therefore, “mAP” is incomplete unless the class averaging and IoU convention are specified. A system may have good overall mAP while failing badly on a safety-critical rare class.

6.2 Segmentation metrics

Pixel accuracy is the fraction of correctly labelled pixels but can be misleading when background dominates. Mean intersection over union is calculated per class and then averaged:

\[

mIoU=\frac{1}{C}\sum_{c=1}^{C}\frac{TP_c}{TP_c+FP_c+FN_c}.

\]

The denominator excludes true negatives for class \(c\), making IoU more informative for imbalanced segmentation. Dice and boundary F-score may be added when overlap or contour quality is important.

6.3 Keypoint metrics

The Percentage of Correct Keypoints (PCK) counts a keypoint as correct when its distance from the target is below a normalized threshold, such as a fraction of person bounding-box size. Object Keypoint Similarity (OKS) is an IoU-like measure that accounts for keypoint distance, object scale, and keypoint-specific annotation uncertainty. A metric must be interpreted together with image resolution, visibility rules, and object scale.

6.4 Operational metrics

Accuracy is only one deployment criterion. Measure latency, throughput, memory, energy, calibration, failure rate, and performance by subgroup and environment. For video, temporal stability and missed-event rate can be more important than frame-level AP.

7. A Practical Implementation Workflow

A reliable project begins with a precise task definition. Decide whether boxes, semantic masks, instance masks, or keypoints are required. Define classes, occlusion rules, image domains, and acceptable errors before collecting data.

Annotations should be audited for inconsistent box tightness, ambiguous boundaries, missing small objects, and keypoint visibility errors. Split data by subject, location, or time when leakage is possible; random image-level splitting can place nearly identical frames in both training and test sets.

A typical training workflow is:

1. Inspect and validate annotations.

2. Create train, validation, and test partitions without leakage.

3. Establish a pretrained baseline.

4. Apply task-appropriate augmentation, such as scale changes, crops, flips, color variation, or mosaic-like composition where valid.

5. Monitor task-specific validation metrics, not only training loss.

6. Tune image resolution, confidence threshold, NMS threshold, learning rate, and loss weights.

7. Analyze errors by class, size, occlusion, lighting, and background.

8. Export and profile the model on target hardware.

9. Test robustness and define a fallback for low-confidence predictions.

Transfer learning is usually effective because pretrained backbones provide general visual features. However, domain shift—such as infrared imagery, surgical images, or unusual camera viewpoints—may require domain-specific pretraining or substantial fine-tuning.

8. Illustrative Case: Warehouse Pallet and Worker Safety

Background

Consider a fictional distribution centre using ceiling cameras to identify pallets, forklifts, and workers. The system must warn when a worker enters a restricted forklift zone and must count pallet locations for inventory management.

Problem

A classification model cannot locate multiple workers or determine whether a person is inside a zone. Bounding boxes can support coarse proximity checks, but a worker’s feet and body boundary may be important near a painted safety line. Pose keypoints may help distinguish standing, bending, and falling, while instance masks can separate overlapping workers.

Analysis

The system has two operational modes. Inventory counting requires object detection, with high recall for pallets. Safety monitoring requires reliable worker detection, low false negatives, and stable tracking. A single aggregate mAP is insufficient because a missed worker is more serious than a duplicate pallet prediction.

Approach

A fast one-stage detector is selected for real-time forklift-camera inference. A second-stage or instance-segmentation model is used offline to audit difficult scenes and generate higher-quality labels. For safety zones, worker detections are projected into the camera’s calibrated ground plane where possible. A pose model estimates lower-body keypoints, but alerts are suppressed when keypoint confidence is low or severe occlusion is present. Temporal tracking requires a worker to remain in the zone for several frames before an alert, reducing one-frame false alarms.

The evaluation reports worker recall, pallet AP, zone-entry false-alarm rate, alert latency, and performance under night lighting and partial occlusion. The team also tests cameras from different warehouse areas rather than reporting only a randomly sampled test set.

Outcome

The combined system provides fast routine inference and a slower audit pipeline. Error analysis shows that increasing image resolution improves small-worker recall more than changing the classifier. The final deployment includes human review for uncertain incidents and retains short, access-controlled video clips rather than unrestricted continuous recordings.

Learning Points

The case demonstrates that architecture selection follows operational requirements. It also shows why detection, segmentation, pose, tracking, calibration, temporal logic, and governance must be designed together. A benchmark score alone cannot establish safety suitability.

9. Challenges, Ethics, and Governance

Occlusion, motion blur, unusual viewpoints, weather, domain shift, and long-tailed class distributions remain major technical challenges. Pixel-level annotation is expensive, and boundary disagreement can limit the meaningful maximum performance. Small objects may contain insufficient visual information, so increasing model complexity cannot solve every failure.

Computer-vision systems can encode demographic and geographic bias when training data underrepresents groups or environments. Performance should be audited by relevant subgroups, with privacy-preserving data practices and documented limitations. Facial, body, workplace, and medical imagery may constitute sensitive personal data under applicable law. Organizations should establish lawful purpose, data minimization, retention limits, access control, security, and mechanisms for human appeal.

Safety-critical deployment requires calibrated confidence, monitoring for distribution shift, fail-safe behavior, version control, incident logging, and human oversight. A detector should not be presented as certain merely because it returns a high score. Model cards, dataset documentation, and reproducible evaluation improve accountability.

10. Contemporary Developments and Best Practices

Current systems increasingly use transformer-based backbones, end-to-end set prediction, open-vocabulary recognition, promptable segmentation, self-supervised pretraining, and lightweight models for edge hardware. End-to-end approaches can reduce dependence on hand-designed post-processing, although they may have different optimization and data requirements. Foundation models can accelerate annotation and adaptation, but their outputs still require verification.

Best practices include:

• Select the task representation that matches the decision, rather than using the most complex model.
• Report class-wise and scale-wise results, not only one average score.
• Keep training, validation, and test conditions representative of deployment.
• Inspect qualitative errors alongside numerical metrics.
• Tune thresholds using validation data and explicit error costs.
• Quantify uncertainty and define behavior for missing or ambiguous predictions.
• Test latency, energy, robustness, and subgroup performance on target hardware.
• Protect sensitive imagery and document data provenance.

Chapter Summary

Object detection predicts classes, locations, and confidence scores; semantic segmentation labels every pixel by class; instance segmentation separates individual objects; and keypoint estimation identifies structural landmarks for pose analysis. One-stage detectors generally favor speed and simpler inference, whereas two-stage detectors explicitly generate and refine regions and often provide strong accuracy at greater computational cost.

Detection systems combine classification and localization losses, overlap measures, and post-processing such as NMS. Segmentation commonly uses pixel-wise cross-entropy, Dice-based objectives, or combinations of losses. Keypoint systems may regress coordinates or predict heatmaps, with visibility and uncertainty handled explicitly. Precision, recall, AP/mAP, mIoU, Dice, PCK, and OKS measure different aspects of performance and must be reported with their conventions.

Successful deployment depends on annotation quality, leakage-free evaluation, error analysis, calibration, target-hardware profiling, privacy, fairness, and safety governance. The appropriate model is determined by the application’s risks and constraints, not by benchmark accuracy alone.

Key Terms

Object detection: Prediction of object classes, locations, and confidence scores.

Bounding box: A rectangular localization of an object.

One-stage detector: A detector that predicts object candidates directly without an explicit proposal stage.

Two-stage detector: A detector that first proposes regions and then classifies and refines them.

Backbone: Feature-extraction network used by a vision model.

Feature pyramid: Multi-resolution feature representation for objects of different scales.

Objectness: Estimated probability that a location or candidate contains an object.

IoU: Intersection area divided by union area between two regions.

NMS: Procedure for suppressing duplicate overlapping detections.

Semantic segmentation: Pixel classification without separating object instances.

Instance segmentation: Pixel-level segmentation that distinguishes individual objects.

Dice coefficient: Overlap measure based on twice the intersection divided by total predicted and target sizes.

Keypoint: A meaningful landmark or structural point.

Heatmap: Spatial probability or confidence map for a keypoint or class.

Pose estimation: Prediction of articulated landmarks or body configuration.

Precision: Fraction of positive predictions that are correct.

Recall: Fraction of actual positives that are detected.

Average precision: Area or summary of a precision–recall curve for a class.

mIoU: Mean of per-class intersection-over-union values.

Object Keypoint Similarity: Scale- and uncertainty-aware keypoint similarity metric.

Review Questions

Short Answer Questions

1. What information does an object detector return that an image classifier ordinarily does not?

2. Distinguish semantic segmentation from instance segmentation.

3. Why are multi-scale features useful in object detection?

4. What problem does non-maximum suppression address?

5. Why can pixel accuracy be misleading in highly imbalanced segmentation data?

Long Answer Questions

1. Compare one-stage and two-stage detectors in terms of architecture, speed, accuracy, and appropriate applications.

2. Explain the roles of classification, objectness, and localization losses in a detector.

3. Describe an encoder–decoder semantic-segmentation system and discuss suitable evaluation metrics.

4. Compare direct coordinate regression and heatmap-based keypoint estimation.

5. Design a complete training and evaluation workflow for an instance-segmentation project.

Analytical / Application Questions

1. A factory needs real-time detection on an edge device but must minimize missed workers. What architecture, metrics, thresholding strategy, and monitoring plan would you recommend? Justify each choice.

2. A segmentation model has 97% pixel accuracy but only 62% mIoU for a rare safety class. Analyze why these results can coexist and propose improvements.

3. A pose system produces unstable knee angles during partial occlusion. Diagnose possible causes and design a solution involving confidence, temporal processing, calibration, and evaluation.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes instance segmentation from semantic segmentation?

A. Instance segmentation predicts only one class for the whole image.

B. Instance segmentation separates different objects that share the same class.

C. Instance segmentation does not require pixel-level predictions.

D. Instance segmentation is limited to human pose estimation.

Correct Answer: B. Instance segmentation separates different objects that share the same class.

Question 2

What is the principal architectural characteristic of a two-stage detector?

A. It uses no feature extractor.

B. It predicts only image-level labels.

C. It generates candidate regions before classification and refinement.

D. It always operates without post-processing.

Correct Answer: C. It generates candidate regions before classification and refinement.

Question 3

If a predicted box and a ground-truth box do not overlap, their ordinary IoU is:

A. 0

B. 0.5

C. 1

D. Undefined in all cases.

Correct Answer: A. 0

Question 4

Why is focal loss useful in dense object detection?

A. It removes the need for labels.

B. It emphasizes easy negative examples.

C. It reduces the influence of easy negatives under class imbalance.

D. It directly computes bounding-box IoU.

Correct Answer: C. It reduces the influence of easy negatives under class imbalance.

Question 5

Which metric averages per-class intersection-over-union values?

A. mIoU

B. PCK

C. Recall

D. Latency

Correct Answer: A. mIoU

Question 6

What does non-maximum suppression usually do?

A. Upsamples the input image.

B. Removes or reduces duplicate overlapping detections.

C. Converts masks into keypoints.

D. Balances the training classes.

Correct Answer: B. Removes or reduces duplicate overlapping detections.

Question 7

A heatmap-based keypoint model primarily predicts:

A. A spatial confidence distribution for each keypoint.

B. Only the image’s dominant class.

C. A single scalar for the entire image.

D. The camera’s focal length.

Correct Answer: A. A spatial confidence distribution for each keypoint.

Question 8

If TP = 80 and FP = 20, what is precision?

A. 0.20

B. 0. fifty

C. 0.80

D. 0.90

Correct Answer: C. 0.80

Question 9

Which metric is specifically associated with scale- and uncertainty-aware keypoint evaluation?

A. OKS

B. mIoU

C. BCE

D. NMS

Correct Answer: A. OKS

Question 10

Why should deployment evaluation include latency and subgroup performance in addition to mAP?

A. Because mAP measures hardware energy exactly.

B. Because benchmark accuracy alone may not reveal operational or fairness failures.

C. Because latency changes the ground-truth labels.

D. Because subgroup performance is unrelated to model behavior.

Correct Answer: B. Because benchmark accuracy alone may not reveal operational or fairness failures.