Robot Sensing, Perception, and Calibration

Introduction

A robot does not act directly on the physical world; it acts through measurements. Encoders estimate joint motion, force–torque sensors reveal interaction, cameras provide appearance and geometry, inertial sensors estimate motion, and range or proximity sensors detect obstacles. These measurements are incomplete, noisy, delayed, and expressed in different coordinate frames. Robotics therefore requires more than selecting sensors: it requires models, calibration, filtering, fusion, and uncertainty assessment.

The central engineering problem is to transform raw measurements into a consistent estimate of robot state and environment. A useful abstraction is

where x_k is the state at time k, z_k is a sensor measurement, h_k(·) is the measurement model, and v_k is measurement noise. The state may include joint position, end-effector pose, velocity, object location, or contact forces. The measurement model depends on sensor physics and robot geometry.

This chapter develops the principles needed to integrate sensing systems in industrial, mobile, and service robots.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain the operating principles, strengths, and limitations of common robot sensors.
2. Distinguish accuracy, precision, resolution, repeatability, bias, and noise.
3. Select suitable filtering and sensor-fusion methods.
4. Formulate coordinate-frame transformations and registration problems.
5. Explain and apply hand–eye calibration.
6. Assess localization and perception uncertainty.
7. Design a sensing architecture for a practical robotic application.
8. Identify safety, privacy, cybersecurity, and ethical issues in robotic perception.

1. Sensor Classes and Measurement Quality

1.1 Proprioceptive and exteroceptive sensing

Proprioceptive sensors measure the robot’s internal state. Joint encoders, motor-current sensors, inertial measurement units (IMUs), and internal temperature sensors are examples. Exteroceptive sensors measure the external world, including cameras, lidar, radar, ultrasonic sensors, tactile arrays, and force–torque sensors.

A sensor can be characterized by:

• Range: the interval over which measurement is valid.
• Resolution: the smallest distinguishable change.
• Accuracy: closeness to the true value.
• Precision: repeatability of measurements.
• Bias: a systematic offset.
• Noise: random variation.
• Latency: time between a physical event and available data.
• Bandwidth: speed at which changing signals can be followed.
• Field of view: angular region observed by a directional sensor.
• Calibration stability: how well the sensor model remains valid over time, temperature, and loading.

A high-resolution encoder may still be inaccurate if its zero offset is wrong. Similarly, a camera may produce detailed images but poor metric depth under weak texture or poor illumination.

1.2 Encoders

Rotary encoders measure joint angle. Incremental encoders generate pulses and require a reference or homing operation; absolute encoders report a coded angular position after power-up. If an encoder has N counts per revolution and quadrature decoding provides a factor q, the nominal angular increment is

where ∆θ is in radians. This is an ideal resolution expression; mechanical backlash, quantization, interpolation error, eccentricity, and transmission compliance may dominate actual accuracy.

For a prismatic joint with a screw lead L metres per revolution,

where ∆d is the nominal linear increment. Joint measurements must be combined with the robot’s kinematic model. If joint offsets or link dimensions are wrong, precise encoder readings can still produce inaccurate end-effector pose.

1.3 Force–torque and tactile sensing

A six-axis force–torque sensor measures a wrench

where forces are measured in newtons and torques in newton-metres. The sensor requires a defined reference point and coordinate frame. Gravitational loading from the tool must be compensated; otherwise a stationary tool can appear to exert a contact force.

Tactile sensors measure distributed contact pressure, shear, vibration, or contact location. A tactile array can estimate a pressure centre:

where p_i is the calibrated pressure at taxel i and x_i,y_i are taxel coordinates. The expression assumes nonnegative, calibrated pressure values and a nonzero total pressure.

Force and tactile sensing support compliant assembly, grasp monitoring, collision detection, slip detection, and manipulation of fragile objects. Limitations include hysteresis, saturation, temperature drift, cross-axis sensitivity, and dependence on contact geometry.

1.4 Proximity, inertial, and range sensors

Inductive and capacitive sensors are effective for short-range detection of particular materials. Ultrasonic sensors are inexpensive and robust to lighting but have wide beams and specular-reflection problems. Infrared time-of-flight sensors can be compact but may be affected by surface reflectivity and ambient radiation.

An IMU combines accelerometers and gyroscopes, and sometimes a magnetometer. An accelerometer measures specific force rather than gravity alone. During dynamic motion, acceleration must be separated from gravity using orientation and a motion model. Gyroscope integration accumulates orientation error because a small bias produces an error that grows with time. IMUs therefore benefit from external observations such as vision, GNSS, or wheel odometry.

Lidar estimates distance using time of flight or phase measurement. For a pulse travelling to a target and back,

where r is range in metres, c is propagation speed, and t is round-trip time. The factor 2 assumes a reflected signal travels the same distance outward and inward. Lidar provides accurate geometry but can fail on transparent, highly absorptive, or strongly reflective surfaces.

1.5 Vision and depth sensing

A pinhole camera model is

where (X,Y,Z) is a 3D point in the camera frame, (u,v) is its image coordinate, f_x,f_y are focal lengths in pixels, and (c_x,c_y) is the principal point. Lens distortion must be included for accurate metrology, commonly through radial and tangential distortion parameters.

Stereo depth depends on disparity:

where B is baseline, f is focal length in pixels, and d is disparity in pixels. Depth uncertainty increases rapidly as disparity becomes small. Structured-light and time-of-flight cameras directly estimate depth but may suffer from sunlight, multipath effects, reflective surfaces, or pattern interference.

Vision supports object recognition, visual servoing, inspection, and scene reconstruction. It is sensitive to illumination, occlusion, motion blur, texture, and viewpoint changes. A reliable system treats image recognition confidence separately from geometric pose uncertainty.

2. Coordinate Frames and Rigid Transformations

Every measurement must be associated with a frame. A point represented in frame A can be transformed to frame B using a homogeneous transformation

where ^B R_A is a 3 × 3 rotation matrix and ^B p_A is the origin of A expressed in B. A point satisfies

using homogeneous point coordinates. Transform composition follows the physical path. For example, a point measured by a camera mounted on a robot flange may be mapped to the robot base by

where B is base, E is end effector, C is camera, and P is the point.

Rotation matrices must satisfy

Errors in frame naming or multiplication order are among the most common causes of apparently mysterious perception failures. A practical convention is to document every frame, axis direction, unit, timestamp, and parent–child relationship in a frame tree.

3. Filtering and State Estimation

3.1 Basic filtering

For a stationary sensor with independent measurements of variance σ², the arithmetic mean of n samples has variance

This reduction assumes independent, identically distributed noise and does not remove bias or motion-induced error. A moving-average filter introduces delay and can blur rapid events.

A first-order discrete low-pass filter is

where 0 < α ≤ 1. Larger α responds faster but smooths less. The filter is useful for noisy scalar signals but should not be applied blindly to force or collision signals where delay can reduce safety.

Median filters reject isolated spikes. Notch filters attenuate known periodic interference. Complementary filters combine signals with different frequency strengths, such as gyroscope orientation at high frequency and accelerometer inclination at low frequency.

3.2 Bayesian estimation

State estimation combines a prior prediction with a measurement. In the linear Gaussian case, the Kalman filter uses

Here F_k is the state-transition matrix, B_k maps control input u_k, Q_k is process-noise covariance, H_k is the measurement matrix, R_k is measurement-noise covariance, K_k is the Kalman gain, and P is state covariance. Superscripts − and + denote before and after measurement update.

The filter assumes an approximately linear model and Gaussian noise. For nonlinear robot kinematics, the extended Kalman filter linearizes the model using Jacobians; the unscented Kalman filter propagates representative sigma points. Particle filters represent non-Gaussian distributions with weighted samples and are useful for ambiguous localization but can be computationally expensive.

3.3 Sensor fusion

Fusion is justified when sensors provide complementary information. Wheel odometry gives smooth short-term motion but drifts. Vision can correct drift but may fail during blur or occlusion. Lidar provides geometric constraints but may be computationally demanding. Fusion should account for correlation: treating two estimates derived from the same underlying data as independent makes uncertainty unrealistically small.

Outlier rejection is essential. A measurement residual is

and its innovation covariance is

A normalized innovation squared test uses

and compares it with a chi-square threshold appropriate to the measurement dimension. This test is meaningful when the model and noise assumptions are approximately valid.

4. Calibration and Registration

4.1 Intrinsic and extrinsic calibration

Intrinsic camera calibration estimates focal lengths, principal point, and distortion. Extrinsic calibration estimates the rigid pose between camera and robot or between multiple sensors. Calibration targets should span the working volume and be viewed at varied orientations; poses concentrated in one plane produce weak geometric excitation.

Robot calibration also estimates joint zero offsets, link dimensions, tool centre point, and compliance parameters. Calibration quality must be evaluated on independent poses rather than only on the poses used for fitting.

4.2 Hand–eye calibration

Hand–eye calibration determines the transform between a robot hand and a camera. In a common formulation, relative robot motions A_i and relative camera motions B_i satisfy

where X is the unknown hand–eye transform. The equation is formed from pairs of poses, not absolute poses, which reduces dependence on the unknown target location. Reliable estimation requires rotational and translational excitation; nearly parallel or tiny motions make the problem poorly conditioned.

A practical procedure is:

1. Rigidly mount the camera and verify mechanical stability.
2. Observe a calibrated target from many robot poses.
3. Record synchronized robot and camera poses.
4. Compute relative motions with consistent frame conventions.
5. Solve the hand–eye equations.
6. Validate by transforming target points through the complete frame chain.
7. Repeat after thermal warm-up and under representative cable loading.

4.3 Frame registration and point-cloud alignment

Registration estimates the transformation between two point sets. Corresponding features can be matched and solved using least squares. Iterative closest point (ICP) alternates between correspondence assignment and transformation estimation. It works well with a good initial alignment and overlapping geometry, but can converge to local minima or incorrect symmetric alignments.

A rigid registration objective is commonly expressed as

subject to R^TR=I and det(R)=1. Here p_i and q_i are corresponding 3D points, R is rotation, and t is translation. The objective assumes correct correspondences and isotropic point errors; weighted variants are preferable when measurement uncertainty differs across points.

5. Uncertainty Assessment

A measurement should be reported with uncertainty, not only a nominal value. For a scalar estimate, a simple model is

where b is bias and n is zero-mean random noise. Repeated measurements estimate random variation but cannot reveal an unknown constant bias without a reference.

For a function y = f(x), small-error covariance propagation uses the Jacobian J:

where Σ_x and Σ_y are covariance matrices. This first-order approximation is appropriate when errors are small and f is locally well approximated by a linear function.

For a robot pose, position and orientation errors are coupled through kinematics. Joint uncertainty can be mapped to Cartesian velocity or small pose perturbation using the manipulator Jacobian J_q:

This expression neglects model-parameter uncertainty and higher-order terms. Near singular configurations, small joint errors may produce large or directionally amplified Cartesian errors.

Uncertainty should be evaluated through repeatability tests, residual analysis, cross-validation, stress testing, and failure-case testing. A perception system that is accurate in laboratory lighting but fails silently in glare is not robust.

6. Integrated Sensing Architecture

A professional robot sensing architecture includes:

• Sensor layer: transducers, signal conditioning, time stamping, and diagnostics.
• Calibration layer: intrinsic, extrinsic, kinematic, and tool models.
• State-estimation layer: filtering, synchronization, and fusion.
• Perception layer: detection, segmentation, recognition, tracking, and geometric estimation.
• Planning and control layer: motion generation, servoing, collision avoidance, and interaction control.
• Safety and supervision layer: limits, watchdogs, plausibility checks, and safe shutdown.

Time synchronization is as important as spatial calibration. If a moving robot uses a camera image captured 100 ms earlier, the apparent camera-to-object geometry is wrong. Hardware triggering, synchronized clocks, and measured latency are preferable to assuming that software arrival time equals acquisition time.

A useful diagnostic strategy is to monitor residuals, sensor saturation, dropped packets, temperature, calibration age, and disagreement between independent sensing modalities. Sensor redundancy should support safe degradation rather than merely duplicate data.

7. Worked Example: Encoder Resolution and Cartesian Uncertainty

An incremental rotary encoder has 2,000 cycles per revolution and is decoded in quadrature, so q = 4. The joint drives a 0.40 m link. Assume independent quantization error with uniform distribution over ±∆θ/2 and ignore all other errors.

The angular increment is

For a small angular error at a 0.40 m radius, the tangential position error is approximately

Thus the increment corresponds to approximately 0.314 mm of tangential motion. This is not a guaranteed end-effector accuracy: link errors, backlash, compliance, calibration error, and other joints may be much larger. The example illustrates the difference between nominal encoder resolution and system-level accuracy.

8. Applied Case: Vision-Guided Robotic Pick-and-Place

Background

Consider an illustrative packaging cell in which a six-axis robot picks randomly oriented components from a conveyor and places them in a fixture. A wrist-mounted RGB-D camera observes the work area, while joint encoders, a wrist force sensor, and a safety lidar provide complementary information.

Problem

Initial trials show missed grasps and occasional fixture collisions. The camera detects components successfully, but placement errors increase when the robot moves rapidly. Force readings also indicate contact before the controller expects it.

Analysis

Investigation identifies four causes: the camera-to-flange transform is inaccurate; image timestamps do not match robot timestamps; depth measurements on dark components are biased; and tool gravity has not been removed from the force signal. The system therefore contains both spatial and temporal calibration errors.

Approach and solution

The team calibrates camera intrinsics using a target at multiple distances and orientations. Hand–eye calibration is performed using well-separated robot motions. The camera clock is synchronized with the robot controller, and measured acquisition latency is included in the pose interpolation. Depth confidence is used to reject unreliable pixels, and a robust point-cloud registration estimates component pose. The force sensor is calibrated with the tool in several orientations to estimate bias and gravitational wrench. During insertion, force is low-pass filtered but a fast threshold path remains active for collision protection.

A fused state estimate combines encoder-based robot pose, camera-based object pose, and force-based contact state. The controller approaches the fixture using vision, slows near the insertion region, and switches to force-guided motion after contact is detected.

Outcome

The principal improvement comes not from a more complex recognition algorithm but from consistent frames, timing, and calibration. The system becomes more tolerant of depth failures because force sensing provides a complementary contact cue. Residual monitoring identifies when camera confidence is too low for autonomous operation.

Learning points

1. Perception performance depends on geometry and timing as well as algorithms.
2. A sensor should be evaluated in the actual operating environment.
3. Fusion is useful when failure modes are complementary.
4. Safety functions should not depend on a single uncertain perception estimate.

9. Limitations, Safety, and Responsible Deployment

Sensors can fail because of occlusion, contamination, electromagnetic interference, vibration, temperature, reflective surfaces, network loss, or adversarial inputs. A robot should detect implausible measurements and enter a defined safe state. Safety-rated sensing and control must follow applicable standards and risk-assessment procedures; ordinary research sensors should not automatically be treated as safety-rated devices.

Cameras and microphones may collect personal data. Deployments should apply data minimization, access control, retention limits, secure communication, and clear notice. Machine-learning perception can encode dataset bias and may be difficult to explain. Confidence scores are not guaranteed probabilities unless properly calibrated. Human operators must know when autonomy is reduced and how to intervene.

Best practices include version-controlled calibration files, traceable units and frame definitions, independent validation data, environmental acceptance tests, fault injection, cybersecurity controls, and maintenance schedules. Calibration should be repeated after mechanical impact, sensor replacement, major temperature changes, or noticeable drift.

10. Contemporary Developments

Modern systems increasingly combine event cameras, neural object detectors, learned depth completion, factor-graph optimization, and differentiable calibration tools. These methods can improve performance but introduce new dependencies on training data, compute hardware, model version, and distribution shift. Learned perception should therefore be integrated with geometric checks, uncertainty estimates, and fallback behavior.

Continuous-time estimation and spatiotemporal calibration address rolling-shutter cameras, moving platforms, and asynchronous sensors. Tactile skins and vision–force policies support dexterous manipulation. Digital twins can compare predicted and measured sensor behavior for maintenance. These advances do not eliminate classical calibration and frame discipline; they make them more important because complex systems contain more interacting assumptions.

Chapter Summary

Robot sensing converts physical phenomena into measurements, but measurements become useful only through models and calibration. Encoders provide joint information; force–torque and tactile sensors reveal interaction; IMUs estimate short-term motion; cameras and depth sensors provide appearance and geometry; lidar, radar, ultrasonic, and proximity sensors support ranging and obstacle detection.

Coordinate frames and homogeneous transformations establish the spatial relationships among sensors, robot links, tools, and objects. Filtering reduces random noise, while Kalman-family methods and particle filters combine models with measurements. Sensor fusion must account for latency, correlation, outliers, and complementary failure modes.

Intrinsic, extrinsic, kinematic, and hand–eye calibration determine whether measurements are geometrically meaningful. Registration aligns coordinate systems or point clouds. Covariance propagation and residual analysis quantify uncertainty, while validation must include realistic operating conditions. Safe and responsible deployment additionally requires diagnostics, secure data handling, privacy protection, and explicit responses to sensor failure.

Key Terms

Sensor fusion: Combining information from multiple sensors to estimate a state.

Proprioception: Measurement of the robot’s internal state.

Exteroception: Measurement of the external environment.

Encoder: Sensor that measures rotary or linear displacement.

Bias: A systematic measurement offset.

Noise: Random variation in a measurement.

Resolution: Smallest nominal distinguishable change.

Force–torque sensor: Sensor measuring three force and three torque components.

Tactile sensor: Sensor measuring contact-related pressure, shear, or vibration.

IMU: Inertial measurement unit containing accelerometers and gyroscopes, and sometimes a magnetometer.

Intrinsic calibration: Estimation of internal sensor parameters such as focal length and distortion.

Extrinsic calibration: Estimation of pose between sensors or between a sensor and robot.

Hand–eye calibration: Estimation of the rigid transform between a robot hand and camera.

Registration: Alignment of measurements or point clouds in a common frame.

Homogeneous transformation: Matrix representing rotation and translation together.

Kalman filter: Recursive estimator for approximately linear Gaussian systems.

Covariance: Matrix describing uncertainty and correlations.

Residual: Difference between an observation and its model prediction.

Latency: Time delay between physical measurement and usable data.

Frame tree: Structured representation of coordinate-frame relationships.

Review Questions

Short Answer Questions

1. Distinguish accuracy from precision and give one robotic example of each.
2. Why must force–torque measurements be associated with a reference frame and reference point?
3. State two limitations of an IMU used without external correction.
4. What is the purpose of hand–eye calibration?
5. Why can timestamp errors produce pose errors even when spatial calibration is correct?

Long Answer Questions

1. Compare encoders, cameras, lidar, force–torque sensors, and IMUs in terms of measured quantity, strengths, and failure modes.
2. Explain the prediction and measurement-update stages of a Kalman filter and state its principal assumptions.
3. Describe a complete calibration procedure for a wrist-mounted RGB-D camera.
4. Explain how coordinate-frame errors, latency, and uncertainty affect vision-guided manipulation.
5. Discuss safety, privacy, and cybersecurity requirements for a perception-enabled industrial robot.

Analytical / Application Questions

1. Design a sensor-fusion architecture for a mobile manipulator operating in a warehouse. Justify sensor selection, frame relationships, filtering method, and fallback behavior.
2. A robot has accurate encoders but its camera-based object placements show a constant 6 mm offset. Propose a systematic diagnostic sequence to distinguish camera calibration, tool calibration, frame-convention, and timing errors.
3. A point-cloud registration system performs well on textured objects but fails on symmetric cylindrical parts. Analyze the cause and propose geometric, sensing, and planning improvements.

Multiple Choice Questions with Answers

Question 1

Which quantity is most directly measured by a rotary encoder?

A. Joint angular position

B. End-effector force

C. Camera depth

D. Surface temperature

Correct Answer: A. Joint angular position

Question 2

For an encoder with N counts per revolution and quadrature factor q, the nominal angular increment is:

A. 2πqN

B. 2π/(qN)

C. qN/(2π)

D. N/(2πq)

Correct Answer: B. 2π/(qN)

Question 3

Which sensor is especially appropriate for measuring contact wrench during robotic insertion?

A. Six-axis force–torque sensor

B. Magnetometer

C. Passive camera lens

D. Incremental encoder on a distant joint

Correct Answer: A. Six-axis force–torque sensor

Question 4

In the pinhole camera model, image coordinates depend directly on which quantity?

A. The ratio of focal length to depth

B. Only the object’s mass

C. The robot battery voltage

D. The lidar pulse repetition rate

Correct Answer: A. The ratio of focal length to depth

Question 5

What is the principal purpose of hand–eye calibration?

A. To estimate the transform between a robot hand and a camera

B. To increase encoder pulse frequency

C. To remove all image noise

D. To determine material hardness

Correct Answer: A. To estimate the transform between a robot hand and a camera

Question 6

Which condition is required of a valid 3 × 3 rotation matrix R?

A. R^T R = I and det(R) = 1

B. R^T R = 0 and det(R) = 0

C. R must be diagonal with positive entries

D. All entries of R must be integers

Correct Answer: A. R^T R = I and det(R) = 1

Question 7

What is a major limitation of integrating gyroscope measurements over time?

A. Bias causes accumulated orientation drift

B. It eliminates all noise automatically

C. It provides direct object identity

D. It requires no initial orientation

Correct Answer: A. Bias causes accumulated orientation drift

Question 8

The covariance propagation expression Σ_y ≈ JΣ_xJ^T is primarily used to:

A. Map small input uncertainties through a function

B. Convert force into electrical power

C. Determine camera pixel colour

D. Calculate encoder counts from voltage only

Correct Answer: A. Map small input uncertainties through a function

Question 9

Why is ICP registration vulnerable when applied to symmetric objects?

A. Different alignments may produce similarly good geometric matches

B. It cannot use any 3D points

C. It always requires a magnetometer

D. Symmetric objects have infinite mass

Correct Answer: A. Different alignments may produce similarly good geometric matches

Question 10

Which practice best supports safe sensor fusion?

A. Monitor residuals, latency, confidence, and sensor-health indicators

B. Assume every sensor is independent and never reject data

C. Use uncalibrated frames to avoid computation

D. Disable diagnostics after commissioning

Correct Answer: A. Monitor residuals, latency, confidence, and sensor-health indicators