Motion Analysis, Tracking, and Video Processing

Introduction and Context

A still image describes appearance at one instant; a video adds time. Motion analysis uses this temporal information to determine what has changed, how objects move, and whether an event is significant. In mechatronics, these capabilities support conveyor inspection, robot navigation, autonomous vehicles, safety monitoring, process control, and human–machine interaction.

A video is commonly represented as a sequence of image frames I_t(x,y), where t is the frame index and (x,y) is the pixel location. If the camera captures at frame rate f_s frames per second, the sampling interval is

where ∆t is measured in seconds. A higher frame rate improves temporal resolution but increases computation, storage, and communication requirements.

Motion processing is not simply a matter of subtracting consecutive images. Illumination changes, camera vibration, shadows, sensor noise, occlusion, perspective, and object deformation all influence the observed image. A successful system therefore combines image formation, temporal filtering, motion estimation, object detection, state estimation, and event logic.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain how video differs from independent image processing.
2. Design temporal filters and frame-difference methods for motion detection.
3. Describe the assumptions and limitations of optical-flow methods.
4. Compare object detection, tracking, and detection–tracking pipelines.
5. Formulate a basic state-space model and apply a Kalman filter.
6. Select performance measures for tracking and event analysis.
7. Design a vision system for conveyor monitoring, navigation, surveillance, or human–machine interaction.
8. Evaluate practical, ethical, and safety issues in deployed video systems.

1. Video as a Spatiotemporal Signal

A video sequence can be modeled as a function of two spatial coordinates and time:

The image intensity may be grayscale or a vector of colour components. Temporal change can be estimated using a forward difference:

For motion detection, the absolute difference is often used:

A binary motion mask is then produced by thresholding:

Here τ is an intensity-difference threshold. The method assumes that the camera is substantially stationary and that background illumination is reasonably stable. It is computationally inexpensive, but it detects all image changes, not only object motion.

The apparent image velocity in pixels per second is related to frame-to-frame displacement by

where d is displacement in pixels between frames. This is image velocity, not necessarily physical velocity. Conversion to metres per second requires camera calibration and a valid geometric model.

Temporal Filtering

Temporal filtering reduces noise and stabilizes measurements. A moving average over N frames is

It reduces independent zero-mean noise, but introduces delay and blurs rapid events. A recursive exponential filter is more memory-efficient:

where 0 < α ≤ 1. A large α responds quickly but smooths less; a small α produces stronger smoothing but greater lag. The filter assumes that the signal changes sufficiently slowly relative to the sampling rate.

For event detection, temporal filtering may be applied to the binary mask. Morphological opening removes isolated noise, while closing fills small gaps. Connected-component labeling converts the mask into candidate regions described by area, bounding box, centroid, and shape.

2. Background Subtraction and Frame Differencing

Frame differencing compares nearby frames and is effective when motion is relatively fast and the background is static. Background subtraction instead estimates a background image B_t(x,y) and compares the current frame with it:

A background model can be updated recursively:

where β controls adaptation. Updating every pixel indiscriminately is dangerous: a stationary foreground object may be absorbed into the background. A practical system updates only pixels classified as background, or uses a mixture-of-Gaussians model when several background states occur, such as waving foliage.

Comparison of Basic Motion Methods

Camera motion violates the stationary-background assumption. Global motion compensation can estimate a transformation, such as a homography, from background feature correspondences and warp one frame into alignment with the next. Residual differences then more closely represent independently moving objects.

3. Optical Flow

Optical flow estimates the apparent velocity of image brightness. Let the image intensity be approximately constant while a small image element moves. The brightness-constancy assumption is

where u and v are horizontal and vertical image velocities. Taking a first-order Taylor approximation gives the optical-flow constraint equation:

Here I_x, I_y, and I_t are partial derivatives of image intensity. This is one equation with two unknowns, so the aperture problem remains: motion cannot be uniquely determined from a single local edge.

The Lucas–Kanade method assumes constant velocity within a small window. For pixels indexed by i in that window, it solves

where each row of A is [I_x,i I_y,i] and b contains I_t,i. A stable solution requires texture in at least two directions; mathematically, AᵀA must be sufficiently well-conditioned. Flat regions and single straight edges produce unreliable estimates.

Dense methods estimate flow at many pixels, whereas sparse methods estimate flow only at selected features such as corners. Pyramidal processing handles larger displacement by estimating motion from coarse to fine image scales. Optical flow is useful for detecting expansion during collision risk, estimating conveyor motion, and recognizing gestures. It is less reliable under severe lighting changes, motion blur, repetitive texture, transparency, or large occlusion.

4. Object Detection and Tracking

Detection answers: “What objects are present, and where?” Tracking answers: “Which object at time t corresponds to which object at time t+1?” A detector may produce a bounding box, class label, confidence score, segmentation mask, or keypoints. A tracker predicts object state between detector updates.

A common tracking-by-detection pipeline is:

1. Acquire and preprocess the frame.
2. Detect candidate objects.
3. Predict the next state of existing tracks.
4. Associate detections with predictions.
5. Update matched tracks.
6. Create tracks for new detections and terminate stale tracks.
7. Apply event rules to the resulting trajectories.

Data association can use distance, intersection over union (IoU), appearance similarity, or a cost matrix. For two bounding boxes, IoU is

It ranges from 0 to 1. An IoU of 1 indicates identical boxes; 0 indicates no overlap. The Hungarian algorithm can find a minimum-cost one-to-one assignment, subject to gating constraints that reject implausible matches.

Track management is essential. A tentative track should normally require several consistent detections before becoming confirmed. A confirmed track may survive a limited number of missed detections, but indefinite survival creates false tracks. Track identity becomes difficult when objects overlap, leave the field of view, or have similar appearance.

Centroid and Region Tracking

For a binary object region containing pixels (x_i,y_i), its centroid is

where n is the number of region pixels. Centroid tracking is appropriate for isolated, compact objects but fails when objects merge or split. Bounding-box tracking retains size information; contour or keypoint tracking may better represent articulated or deformable objects.

5. State Estimation with the Kalman Filter

Measurements are noisy and may be temporarily absent. State estimation combines a motion model with observations. A simple constant-velocity state is

For sampling interval ∆t, the discrete state model is

with

The measurement model may observe position only:

Here w_t and v_t are process and measurement noise, usually modeled as zero-mean random variables with covariance matrices Q and R. The model assumes approximately linear dynamics and Gaussian noise.

The prediction step is

The innovation and its covariance are

The Kalman gain, state update, and covariance update are

The covariance P expresses uncertainty. Large R means measurements are unreliable, so the filter trusts prediction more. Large Q means the motion model is uncertain, so the filter responds more strongly to measurements. For nonlinear motion or measurements, extended or unscented Kalman filters may be used; particle filters represent non-Gaussian, multimodal uncertainty.

Worked Example: One-Dimensional Prediction and Update

Suppose a conveyor object has predicted position 10 cm and predicted variance 4 cm². A camera measures 12 cm with measurement variance 1 cm². The scalar Kalman gain is

The updated position is

The updated variance is

The estimate lies closer to the measurement because the camera measurement is more precise than the prediction. This scalar example assumes a linear model and independent Gaussian errors.

6. Video Event Analysis

Tracking produces trajectories; event analysis converts them into decisions. Common events include entering a region, crossing a line, stopping, reversing direction, collision risk, loitering, queue formation, and unsafe human–robot proximity.

A line-crossing event can be detected from the sign change of an object's signed distance to a line. If a line is represented by

then the signed value is s_t = a x_t + b y_t + c. A crossing occurs when s_t s_t−1 < 0, subject to a minimum displacement and temporal confirmation. Hysteresis prevents repeated triggers caused by jitter: use one threshold to activate and a different threshold to reset.

For speed estimation, a calibrated ground-plane transformation is required. If image coordinates are mapped to world coordinates (X,Y), then

This approximation assumes the sampling interval is short and that corresponding points lie on a calibrated plane. Pixel displacement alone is not a reliable physical speed measurement when depth varies.

For a conveyor, event logic may include: object detected, object accepted or rejected, object missing, object jammed, and object too close to its predecessor. The logic should include debounce intervals, confidence thresholds, and safe failure states rather than reacting to one uncertain frame.

7. System Architecture and Engineering Design

A practical motion-vision system includes illumination, camera, lens, triggering, processing hardware, communication, and an actuator or supervisory interface. Global-shutter cameras are preferred for fast moving objects because rolling shutter exposes different image rows at different times and can distort shape.

Important design variables include:

A useful timing budget is

where each term is a latency component. If an object travels at physical speed v during total latency, the object moves approximately

The actuation point must therefore be placed sufficiently far downstream, or prediction must compensate for delay. Units must be consistent: if v is in m/s and T_total in seconds, d_latency is in metres.

8. Applied Case: Illustrative Conveyor Monitoring System

Background

A packaging line transports boxes at approximately 0.8 m/s. A camera must count boxes, identify gaps, detect jams, and signal a reject actuator. The camera views a fixed section of conveyor from above.

Problem

Simple frame differencing generates false alarms from illumination variation and vibration. Boxes may briefly overlap in the image, and the actuator must operate at the correct downstream position.

Analysis

The conveyor is nearly planar and the camera is stationary, so background modeling and planar calibration are suitable. At 30 frames/s, the sampling interval is

During one interval, a box traveling at 0.8 m/s moves approximately

or 2.66 cm. If the box image moves 18 pixels per frame, the image speed is 540 pixels/s, but that value is not used as physical speed without calibration.

Approach and Solution

The system uses diffuse strobed illumination, a global-shutter camera, a calibrated image-to-conveyor-plane mapping, and a running background model. A lightweight object detector initializes tracks. A Kalman filter predicts centroid position between detections. IoU and predicted-position gating associate detections. A box crossing a virtual inspection line is counted once using a confirmed-track identifier.

A jam event is declared only when a confirmed track remains within a narrow displacement range for a specified time, while upstream objects continue to arrive. A missing-box event is generated when the expected time window passes without a confirmed crossing. The reject command is scheduled using the measured conveyor coordinate and total control latency.

Outcome

The combined system is more robust than raw frame subtraction because the background model handles gradual illumination change, tracking bridges short detector failures, and event hysteresis prevents duplicate counts. The design still requires periodic recalibration, monitoring of camera cleanliness, and a safe response when confidence falls below an operating threshold.

Learning Points

1. Motion detection, tracking, and event interpretation are distinct tasks.
2. Calibration is necessary for metric speed and position.
3. Latency directly affects actuator timing.
4. Track confirmation and hysteresis reduce false events.
5. A vision system should report uncertainty and fail safely.

9. Evaluation, Limitations, and Responsible Deployment

Performance should be evaluated on representative video, including difficult conditions. Detection metrics include precision, recall, and IoU. Tracking metrics may include identity switches, track fragmentation, localization accuracy, and time-to-detection. For an event detector, false alarms and missed events should be reported separately because their costs may differ greatly.

Important failure sources include poor exposure, motion blur, insufficient texture, camera shake, shadows, reflections, occlusion, object similarity, domain shift, and network latency. A model trained in daylight may fail at night; a detector trained on one conveyor arrangement may fail after packaging changes.

Video systems raise governance issues. Surveillance should have a defined purpose, limited retention, access control, encryption, and appropriate notice. Biometric identification and workplace monitoring may be regulated differently from anonymous object counting. Training data can encode demographic bias, and automated decisions affecting workers require human oversight and documented appeal processes. Safety-related systems should not rely on an unvalidated vision model as the sole protective measure; independent interlocks and conventional safety devices may be required.

Contemporary systems increasingly combine deep neural detectors, transformer-based video models, event cameras, edge accelerators, and sensor fusion. These technologies can improve performance but introduce computational, cybersecurity, explainability, and maintenance challenges. The engineering principle remains unchanged: select the simplest validated method that satisfies the required accuracy, latency, safety, and cost.

Best-Practice Checklist

• Stabilize or compensate the camera before interpreting motion.
• Choose frame rate and exposure from the fastest relevant event.
• Use controlled illumination whenever possible.
• Calibrate the camera if physical distance or speed matters.
• Separate detection, prediction, association, and event logic.
• Tune process and measurement uncertainty using validation data.
• Use track confirmation, gating, and hysteresis.
• Test under occlusion, lighting change, vibration, and communication delay.
• Log confidence, timing, and failures for diagnosis.
• Protect privacy and provide a safe fallback for uncertain outputs.

Chapter Summary

Motion analysis treats video as a time-varying signal rather than a collection of unrelated images. Frame differencing and background subtraction provide efficient methods for detecting change, while temporal filters reduce noise at the cost of delay. Optical flow estimates apparent image motion under brightness and small-motion assumptions, but suffers from the aperture problem and weak texture.

Object detection identifies objects; tracking maintains their identities and estimates their trajectories. Data association, track management, and occlusion handling are central to reliable systems. The Kalman filter combines a dynamic model with noisy measurements and provides both state estimates and uncertainty. Event analysis applies geometric and temporal rules to trajectories for counting, speed estimation, jam detection, safety monitoring, and interaction.

Engineering deployment requires attention to frame rate, exposure, calibration, latency, illumination, computation, and safety. Evaluation must include missed events, false alarms, identity errors, and difficult operating conditions. Ethical deployment requires purpose limitation, privacy protection, human oversight, and secure handling of video data.

Key Terms

Temporal filtering: Processing a sequence over time to reduce noise or extract changing components.

Frame differencing: Estimating motion by subtracting successive video frames.

Background subtraction: Comparing a current frame with an estimated scene background.

Motion mask: A binary image indicating pixels classified as moving or changed.

Optical flow: Apparent image velocity estimated from brightness patterns.

Brightness constancy: Assumption that a moving image point retains approximately constant intensity.

Aperture problem: Ambiguity in estimating motion from a locally observed edge.

Object detection: Locating and classifying objects in an image.

Tracking: Maintaining an object's state and identity over time.

Data association: Matching current detections to existing tracks.

Intersection over union: Overlap area divided by union area for two regions or boxes.

Kalman filter: Recursive estimator for approximately linear systems with Gaussian uncertainty.

State vector: Variables describing an object's position, velocity, and related properties.

Process noise: Uncertainty in the assumed dynamic model.

Measurement noise: Uncertainty in sensor observations.

Occlusion: Partial or complete blocking of an object by another object.

Hysteresis: Use of different activation and reset thresholds to prevent jitter.

Global shutter: Camera exposure method that captures the image simultaneously.

Latency: Time between sensing an event and producing the corresponding action.

Event analysis: Interpretation of trajectories and observations as meaningful occurrences.

Review Questions

Short Answer Questions

1. Why does frame differencing detect illumination changes as well as object motion?
2. State the optical-flow constraint equation and identify its variables.
3. What is the purpose of data association in multi-object tracking?
4. Why is calibration required for converting image motion into physical speed?
5. How do process noise and measurement noise affect a Kalman filter?

Long Answer Questions

1. Compare frame differencing, background subtraction, optical flow, and tracking-by-detection for an indoor monitoring application.
2. Explain the prediction and update stages of the Kalman filter for a constant-velocity tracker.
3. Design a video-processing pipeline for counting objects on a conveyor and discuss likely failure modes.
4. Explain how camera motion, shadows, and occlusion affect motion detection and propose mitigation methods.
5. Discuss privacy, safety, and governance requirements for a workplace video-monitoring system.

Analytical / Application Questions

1. A camera operates at 25 frames/s while an object moves at 1.2 m/s. Calculate the distance traveled between successive frames and explain its implication for motion blur and tracking.
2. A predicted object position is 50 pixels with variance 9 pixels². A measurement is 56 pixels with variance 4 pixels². Using the scalar Kalman update, calculate the gain and updated position.
3. Design an event rule that detects a conveyor jam while avoiding false alarms caused by one missed detection. Specify the required track states, timing logic, and reset conditions.

Multiple Choice Questions with Answers

Question 1

What is the main assumption behind basic frame differencing for motion detection?

A. The camera and background are reasonably stable

B. Every object has a unique colour

C. The scene contains no sensor noise

D. All motion is perpendicular to the image plane

Correct Answer: A. The camera and background are reasonably stable

Question 2

In the optical-flow equation I_xu + I_yv + I_t = 0, what do u and v represent?

A. Image intensity and exposure

B. Horizontal and vertical image velocities

C. Object width and height

D. Process and measurement noise

Correct Answer: B. Horizontal and vertical image velocities

Question 3

An IoU value of 1 between two bounding boxes means that the boxes are

A. Adjacent but non-overlapping

B. Identical in area and location

C. Different sizes with one inside the other

D. Separated by one pixel

Correct Answer: B. Identical in area and location

Question 4

Which condition is required for a basic constant-velocity Kalman model to be directly appropriate?

A. Approximately linear motion and a suitable noise model

B. Perfectly known object acceleration

C. No measurements after initialization

D. A stationary object in every frame

Correct Answer: A. Approximately linear motion and a suitable noise model

Question 5

At 30 frames/s, what is the approximate sampling interval?

A. 0.0033 s

B. 0.0333 s

C. 0.30 s

D. 30 s

Correct Answer: B. 0.0333 s

Question 6

If predicted variance is 4 and measurement variance is 1 in a scalar Kalman filter, the gain is

A. 0.20

B. 0.25

C. 0.80

D. 4.00

Correct Answer: C. 0.80

Question 7

Which technique helps prevent repeated event triggers caused by measurement jitter?

A. Reducing all image resolution

B. Hysteresis and temporal confirmation

C. Removing the tracker

D. Increasing camera vibration

Correct Answer: B. Hysteresis and temporal confirmation

Question 8

Why is a global-shutter camera often preferred for fast conveyor inspection?

A. It captures all image rows at the same time

B. It eliminates the need for illumination

C. It always provides metric calibration

D. It guarantees perfect detection

Correct Answer: A. It captures all image rows at the same time

Question 9

A conveyor object moves at 0.8 m/s and the frame interval is 0.0333 s. Approximately how far does it move between frames?

A. 0.00266 m

B. 0.0266 m

C. 0.266 m

D. 2.66 m

Correct Answer: B. 0.0266 m

Question 10

What is the principal purpose of track confirmation in a tracking system?

A. To accept every single noisy detection immediately

B. To reduce false tracks caused by transient detections

C. To remove all missed detections permanently

D. To convert pixels directly into metres

Correct Answer: B. To reduce false tracks caused by transient detections