Robot Perception, Localization and Mapping

Introduction

A robot can act intelligently only if it can estimate what exists around it, where it is, and how its surroundings are structured. These capabilities are collectively known as robot perception, localization, and mapping. Perception converts sensor measurements into useful representations such as edges, objects, surfaces, and landmarks. Localization estimates the robot’s pose relative to a reference frame. Mapping constructs a representation of the environment. When the robot must estimate its pose while simultaneously constructing an unknown map, the problem is called simultaneous localization and mapping (SLAM).

These functions are central to autonomous mobile robots, warehouse vehicles, agricultural machines, inspection systems, service robots, autonomous cars, and unmanned aerial vehicles. They are also difficult because measurements are incomplete, noisy, time-varying, and affected by illumination, occlusion, reflections, wheel slip, calibration errors, and dynamic objects.

A useful perception system therefore does more than detect objects. It maintains uncertainty, combines multiple sensors, rejects implausible measurements, and selects representations appropriate to the task.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain the relationship among sensing, perception, localization, mapping, and planning.
2. Compare cameras, depth cameras, LiDAR, inertial sensors, wheel encoders, and sonar.
3. Apply basic image-processing and point-cloud-processing methods.
4. Extract and match visual and geometric features.
5. Formulate robot motion and observation models.
6. Use Bayesian estimation, Kalman filtering, extended Kalman filtering, particle filtering, and pose-graph optimization.
7. Explain occupancy-grid mapping, landmark mapping, and SLAM.
8. Evaluate the effects of noise, uncertainty, calibration, outliers, and data association.
9. Select suitable algorithms for practical robotic applications.

1. Perception as an Estimation Problem

A robot does not observe the world directly. A sensor produces a measurement that depends on the environment, the robot’s pose, the sensor model, and random error. Let the robot state at time t be [[M:x_t]], the control input be [[M:u_t]], and the measurement be [[M:z_t]]. A general state-space model is:

Here, [[M:f]] is the motion model, [[M:h]] is the measurement model, [[M:m]] is the map or environmental state, [[M:w_t]] is process noise, and [[M:v_t]] is measurement noise. The noise terms represent uncertainty rather than physical forces alone. They may arise from wheel slip, timing errors, sensor quantization, inaccurate models, or imperfect calibration.

The objective is to estimate the posterior probability:

This is the probability distribution of the current state and map given all measurements and controls up to time t. A point estimate, such as the most likely pose, is useful, but the distribution is more informative because it expresses confidence and ambiguity.

Coordinate Frames and Pose

A mobile robot commonly uses:

• a world or map frame, fixed to the environment;
• a robot or body frame, attached to the robot;
• a sensor frame, attached to a camera, LiDAR, or other sensor;
• an odometry frame, representing locally integrated motion.

In a planar environment, pose is often represented as [[M:x = (x, y, θ)]], where x and y are position in metres and [[M:θ]] is heading in radians. In three dimensions, pose consists of a translation vector and a rotation. Homogeneous transformation matrices provide a compact representation:

where [[M:R]] is a 3 × 3 rotation matrix and [[M:t]] is a 3 × 1 translation vector. Transformations must be composed in the correct order because three-dimensional rotations do not generally commute.

2. Sensors for Robot Perception

No sensor provides a complete and perfect description of the environment. Sensor selection is therefore a design decision involving range, resolution, field of view, update rate, cost, environmental sensitivity, and computational demand.

Calibration and Synchronization

Calibration determines intrinsic parameters, such as focal length and principal point, and extrinsic parameters, such as the rigid transformation between a camera and a robot body. An uncalibrated sensor may produce apparently plausible but geometrically incorrect results. Time synchronization is equally important. If a robot moves rapidly while camera and IMU timestamps differ, the measurements may correspond to different poses.

Sensor fusion is effective only when the sensors are calibrated, synchronized, and expressed in compatible coordinate frames.

3. Computer Vision for Robotics

Image Formation and Preprocessing

For a pinhole camera, a three-dimensional point [[M:(X,Y,Z)]] projects approximately to pixel coordinates:

where [[M:f_x]] and [[M:f_y]] are focal lengths in pixel units, [[M:(c_x,c_y)]] is the principal point, and Z is depth along the optical axis. This model assumes a calibrated camera and ignores lens distortion. Real lenses require radial and tangential distortion correction.

Preprocessing may include resizing, denoising, contrast adjustment, colour-space conversion, rectification, and masking. Processing should be task-dependent. Excessive smoothing can remove edges needed for localization, while aggressive contrast enhancement can amplify noise.

Features and Descriptors

A feature is a repeatable image structure that can be detected across changes in viewpoint or illumination. Common feature types include corners, blobs, edges, and line segments. A detector identifies feature locations; a descriptor encodes the local appearance for matching.

Examples include Harris corners, FAST, SIFT, SURF, ORB, and binary descriptors. ORB is often attractive in embedded robotics because it provides relatively efficient detection and matching. Feature quality should be assessed by repeatability, distinctiveness, spatial distribution, and computational cost.

Feature matching usually involves comparing descriptors. For a descriptor a, the nearest and second-nearest candidate distances may be [[M:d_1]] and [[M:d_2]]. A ratio test accepts a match only when:

where [[M:τ]] is a selected threshold, commonly less than 1. The exact threshold depends on the descriptor and application. Matching must also be checked with geometric verification because similar-looking regions can produce false correspondences.

Robust Geometric Estimation

Outliers are common in visual matching. Random Sample Consensus (RANSAC) repeatedly selects a minimal sample, estimates a model, and counts measurements consistent with that model. For a homography, the model may explain image-plane transformations; for localization, an essential matrix, fundamental matrix, or perspective-n-point model may be appropriate.

RANSAC is robust when the proportion of inliers is sufficiently high and the inlier error threshold reflects sensor resolution and modelling error. It does not guarantee a correct model if the scene is degenerate, such as a largely planar scene viewed with insufficient motion.

4. Point-Cloud Processing

A point cloud is a set of three-dimensional points, commonly represented as [[M:p_i = (x_i,y_i,z_i)]]. Point clouds may be produced by LiDAR, stereo, structured light, or depth cameras.

A typical processing pipeline is:

1. Transform points into a common reference frame.
2. Remove invalid points and obvious range outliers.
3. Apply voxel-grid downsampling if computational reduction is required.
4. Estimate surface normals where geometry is needed.
5. Segment planes, clusters, or objects.
6. Register successive clouds or compare them with a map.

Filtering

A statistical outlier filter can reject points whose neighbourhood distance is unusually large. A radius filter retains a point only when enough neighbours occur within a specified radius. Filtering parameters must reflect point density; a threshold suitable for a dense depth camera may incorrectly delete valid sparse LiDAR returns.

Segmentation and Registration

Plane fitting can identify floors, walls, or tabletops. A plane may be expressed as:

where the normal vector is [[M:n=(a,b,c)]]. The coefficients are defined up to a nonzero scale, and a normalized normal is often used for interpreting distances.

Point-cloud registration estimates the rigid transformation aligning two clouds. Iterative Closest Point (ICP) alternates between associating corresponding points and minimizing alignment error. A point-to-point objective is:

where [[M:p_i]] and [[M:q_i]] are corresponding points, [[M:R]] is a rotation, and [[M:t]] is a translation. ICP is local: it requires a sufficiently good initial estimate and can fail in symmetric, featureless, or heavily overlapping environments. Variants use point-to-plane error, normals, robust losses, or global descriptors.

5. Probabilistic Localization

Localization estimates pose relative to a known map or reference. Dead reckoning integrates motion but accumulates drift. Absolute observations, such as landmarks or map features, can reduce drift, although they may be ambiguous.

Bayesian Filtering

Bayesian filtering consists of prediction and correction. The prediction step uses the motion model:

The correction step incorporates the new measurement:

These equations assume that the motion and measurement models adequately describe the system. The integral is often computationally intractable, motivating approximate filters.

Kalman Filter

The standard Kalman filter applies to linear models with additive, zero-mean Gaussian noise. For:

where [[M:w_t ∼ N(0,Q_t)]] and [[M:v_t ∼ N(0,R_t)]], the prediction equations are:

The correction equations are:

Here, [[M:P]] is covariance, [[M:K]] is the Kalman gain, and [[M:I]] is the identity matrix. Covariance has squared units: position variance is in m², while angular variance is in rad².

Extended Kalman Filter

Robot motion and observations are generally nonlinear. The Extended Kalman Filter (EKF) linearizes the models around the current estimate. If [[M:F_t]] is the Jacobian of the motion model and [[M:H_t]] is the Jacobian of the observation model, the covariance prediction uses [[M:F_t P F_tᵀ + Q_t]]. Linearization can be inaccurate for large uncertainty, highly nonlinear motion, or poor initial estimates.

Particle Filter

A particle filter represents a distribution using weighted samples. Each particle contains a possible pose. The algorithm samples from the motion model, weights particles according to measurement likelihood, normalizes weights, and resamples. It handles non-Gaussian and multimodal beliefs, such as two possible corridors, but can suffer from particle depletion and requires sufficient particles.

6. Mapping Representations

Occupancy Grids

An occupancy grid divides space into cells, each storing the probability that the cell is occupied. A log-odds representation is useful because independent evidence can be accumulated additively:

Here, [[M:l_t]] is the cell’s log odds, [[M:p(m_i|z_t,x_t)]] is the inverse sensor-model probability, and [[M:l_0]] is the prior log odds. The model assumes conditional independence approximations and requires correct ray tracing. Occupancy grids are intuitive for navigation but consume memory at high resolution.

Feature and Landmark Maps

A landmark map stores entities such as corners, poles, signs, or reflective targets, together with their estimated positions and uncertainty. It is compact and useful when landmarks are distinctive, but it depends on reliable detection and data association.

Topological and Semantic Maps

A topological map represents places and their connectivity rather than precise metric geometry. A semantic map adds meaning, such as “charging station,” “door,” or “workbench.” Metric, topological, and semantic representations can be combined: a robot may use metric geometry for collision avoidance and semantic labels for task planning.

7. Simultaneous Localization and Mapping

SLAM is required when neither the robot pose nor the map is initially known. The central difficulty is circular: accurate mapping requires localization, while accurate localization requires a map.

A classical landmark-based SLAM state may be written as:

The estimator updates robot and landmark states jointly. Correlations are important: if the robot revisits a previously observed landmark, the new observation can reduce uncertainty in both the current pose and earlier poses.

Front End and Back End

A SLAM system has two conceptual components. The front end processes sensor data, extracts features or scans, estimates relative motion, and proposes loop closures. The back end optimizes poses and landmarks subject to measurement constraints.

In pose-graph SLAM, each node is a robot pose and each edge is a relative-pose constraint. A least-squares objective is:

where [[M:e_ij]] is the residual for an edge between poses i and j, [[M:Ω_ij]] is the information matrix, and [[M:X]] is the set of poses. The information matrix is the inverse covariance when that inverse exists and is correctly defined. Robust loss functions are often added to reduce the influence of incorrect loop closures.

Loop Closure

Loop closure detects that the robot has returned to a previously visited location. Without loop closure, odometry and local registration drift accumulate. A correct loop closure can distribute the correction over a trajectory; an incorrect one can severely distort the map. Therefore, loop closures require appearance or geometric verification, not merely a nearest-neighbour database match.

Visual, LiDAR, and Visual-Inertial SLAM

Visual SLAM extracts motion and structure from images. It may fail in darkness, low texture, motion blur, or scenes dominated by dynamic objects. LiDAR SLAM uses geometric scans and is often more robust to illumination, but may struggle in repetitive corridors or geometrically sparse environments. Visual-inertial SLAM combines camera measurements with IMU dynamics, improving short-term motion estimation and scale observability. Sensor fusion increases capability but also increases calibration and synchronization requirements.

8. Uncertainty, Data Association, and Evaluation

Sources of Uncertainty

Uncertainty may be random, systematic, or epistemic. Random noise varies from measurement to measurement. Systematic error includes biased encoders, lens distortion, and misaligned sensors. Epistemic uncertainty reflects incomplete knowledge, such as an unobserved obstacle or an unmodelled dynamic object.

A covariance matrix describes uncertainty under a local Gaussian approximation. Off-diagonal terms represent correlation. Ignoring correlation can produce overconfident estimates and unsafe decisions.

Data Association

Data association determines which observation corresponds to which landmark or map feature. A common consistency test uses the innovation:

and its covariance:

The squared Mahalanobis distance is:

A measurement is accepted only when this value is consistent with a selected statistical gate. The threshold depends on the measurement dimension and desired confidence; it should not be chosen arbitrarily.

Performance Measures

Relevant measures include localization root-mean-square error, absolute trajectory error, relative pose error, map accuracy, loop-closure precision and recall, latency, memory use, and failure rate. A system that is accurate but too slow for real-time control may be unsuitable. Evaluation should include unseen environments and adverse conditions rather than only favourable laboratory scenes.

9. Worked Example: One-Dimensional Sensor Fusion

Consider a robot moving along a straight rail. After prediction, its position estimate is 10.0 m with variance 4.0 m². A range sensor reports 12.0 m with variance 1.0 m². Assume a linear observation model, independent zero-mean Gaussian errors, and no bias.

The scalar Kalman gain is:

The innovation is 2.0 m, so the updated position is:

The updated variance is:

The result lies closer to the more precise sensor measurement, while uncertainty decreases from 4.0 m² to 0.8 m². This example is one-dimensional and linear; a real robot commonly requires nonlinear motion, multidimensional covariance, and outlier handling.

10. Illustrative Case: Indoor Warehouse Robot

Background

An illustrative warehouse robot transports containers between storage aisles and a packing station. The building contains long repetitive corridors, reflective shelves, changing lighting, pedestrians, and occasional pallet obstructions.

Problem

Wheel odometry alone drifts substantially during long journeys. A camera is unreliable in poorly illuminated aisles, while a two-dimensional LiDAR can detect shelving but may produce ambiguous scans in repetitive corridors. The robot must localize safely and update its map without treating moving workers as permanent obstacles.

Analysis

The system uses wheel encoders and an IMU for high-rate prediction, LiDAR for geometric localization, and a camera for semantic recognition of signs and station labels. Static structure is separated from short-lived obstacle detections. A pre-existing occupancy map supports global localization, while a local cost map is updated continuously for obstacle avoidance.

Approach and Solution

The estimator propagates pose using a motion model driven by wheel and inertial measurements. Scan matching supplies relative geometric corrections. A pose-graph back end maintains selected keyframes and verifies loop closures using both geometric alignment and appearance information. The robot uses a particle filter during global relocalization because several corridors may initially be plausible. Once a unique location is identified, a local Gaussian estimator provides efficient tracking.

Sensor extrinsics are calibrated on a rigid fixture, timestamps are synchronized, and diagnostic software monitors innovation statistics. Measurements with implausibly large residuals are rejected or down-weighted. Dynamic objects are represented in a temporary obstacle layer rather than inserted into the permanent structural map.

Outcome

The combined system reduces odometric drift, remains operational under moderate lighting variation, and avoids corrupting the long-term map with pedestrians. Its main residual risks are glass surfaces, severe LiDAR occlusion, incorrect loop closures, and failures caused by major changes in the warehouse layout.

Learning Points

1. No single sensor is adequate for all warehouse conditions.
2. A permanent map and a dynamic obstacle map should be distinguished.
3. Redundant sensing improves robustness only when calibration and timing are controlled.
4. Relocalization and local tracking may require different estimation methods.
5. Safety depends on detecting estimator failure, not merely on obtaining a pose estimate.

11. Implementation Practices and Limitations

A dependable implementation should begin with a clear frame convention, calibrated sensors, recorded datasets, and repeatable evaluation. Every algorithm should expose diagnostics such as tracking confidence, covariance, rejected measurements, map growth, processing latency, and loop-closure decisions.

Important limitations include:

• Observability: Some motions or environments do not reveal all state variables. A monocular camera, for example, does not directly determine absolute metric scale without additional information.
• Degeneracy: Repeated textures, planar scenes, symmetry, and featureless walls create ambiguity.
• Dynamic environments: Standard static-world SLAM assumptions are violated by people, vehicles, and movable objects.
• Drift and bias: Integrating biased velocity or acceleration produces growing error.
• Computational constraints: High-resolution maps and dense point clouds may exceed embedded processing or memory limits.
• Model mismatch: Incorrect noise models can make an estimator either unstable or falsely confident.
• Security: Adversarial visual patterns, spoofed landmarks, or malicious sensor inputs may mislead perception.

12. Contemporary Developments and Responsible Deployment

Modern systems increasingly combine classical geometric estimation with learned components. Neural networks can provide object detection, depth estimation, place recognition, feature description, and semantic segmentation. However, learned outputs remain uncertain and may fail under domain shift—for example, when training data contain daylight scenes but deployment occurs at night.

Best practice is to use learning where it provides measurable benefit while retaining geometric verification, uncertainty monitoring, and safety fallbacks. Dataset documentation, testing across environments, protection of recorded images, and human oversight are important governance measures. In workplaces, the robot must not only localize accurately but also respect privacy, access control, safety zones, and applicable machinery and workplace-safety standards.

A practical architecture separates safety-critical collision prevention from experimental perception modules. Emergency stopping should not depend exclusively on a complex learned classifier or an unverified loop closure. System logs should support failure analysis without unnecessarily collecting personal information.

Chapter Summary

Robot perception converts sensor data into geometric, visual, and semantic information. Localization estimates robot pose, while mapping represents the environment. Both are estimation problems because sensors and motion models are uncertain. Cameras provide rich appearance information; LiDAR provides geometric range; IMUs and encoders provide high-rate motion information but drift. Calibration and synchronization are prerequisites for reliable fusion.

Computer vision commonly uses preprocessing, feature detection, descriptor matching, geometric verification, and robust estimation. Point-cloud processing includes filtering, segmentation, normal estimation, and registration. Bayesian filtering provides the general framework for prediction and correction. Kalman filters are efficient for linear-Gaussian systems, EKFs handle moderate nonlinearities through linearization, and particle filters represent non-Gaussian or multimodal beliefs.

Occupancy grids, landmark maps, pose graphs, topological maps, and semantic maps serve different purposes. SLAM couples localization and mapping and relies on front-end data association, back-end optimization, and reliable loop closure. Uncertainty, observability, dynamic objects, calibration, outliers, and computation determine practical performance. Robust deployment requires quantitative evaluation, failure monitoring, responsible data handling, and safety mechanisms independent of uncertain perception.

Key Terms

Perception: The process of converting sensor data into information useful for robot action.

Localization: Estimation of a robot’s pose relative to a reference frame or map.

Mapping: Construction of a representation of the robot’s environment.

SLAM: Simultaneous localization and mapping in an initially unknown environment.

Pose: Position and orientation of a robot or sensor.

Coordinate frame: A reference system used to express positions, orientations, and transformations.

Calibration: Estimation of sensor parameters and relative sensor-to-body transformations.

Point cloud: A collection of three-dimensional sample points.

Feature descriptor: A numerical representation used to compare local image or geometric structures.

RANSAC: A robust model-fitting method that separates likely inliers from outliers.

ICP: An iterative algorithm for aligning overlapping point clouds.

Bayesian filter: A recursive estimator that combines prediction and measurement likelihood.

Kalman filter: An optimal linear-Gaussian recursive estimator under its stated assumptions.

Particle filter: A sampling-based estimator for arbitrary or multimodal probability distributions.

Occupancy grid: A spatial grid whose cells store estimated occupancy probabilities.

Data association: Determination of which measurement corresponds to which landmark or map element.

Loop closure: Recognition that the robot has returned to a previously visited place.

Covariance: A matrix describing uncertainty and correlations among estimated variables.

Pose graph: A graph whose nodes are poses and whose edges are relative-pose constraints.

Observability: The extent to which system states can be inferred from available inputs and measurements.

Review Questions

5 Short Answer Questions

1. Why is uncertainty an essential part of robot localization?
2. Distinguish a sensor frame from a world frame.
3. What is the purpose of RANSAC in feature matching?
4. State one strength and one limitation of occupancy-grid mapping.
5. Why can wheel odometry drift over time?

5 Long Answer Questions

1. Explain the prediction and correction stages of Bayesian filtering and relate them to robot localization.
2. Compare Kalman filters, EKFs, and particle filters in terms of model assumptions and applications.
3. Describe a complete point-cloud-processing pipeline from raw measurements to registration.
4. Explain the roles of the front end, back end, and loop closure in a SLAM system.
5. Discuss how calibration, synchronization, dynamic objects, and outliers affect a multisensor robot.

3 Analytical / Application Questions

1. A robot has accurate local odometry but repeatedly loses localization in visually repetitive corridors. Propose a sensor-fusion and map-management strategy, and justify your choices.
2. Design an evaluation plan for a warehouse SLAM system. Identify datasets or operating conditions, performance measures, failure cases, and safety diagnostics.
3. A camera-based robot performs well in training environments but fails at night and in rain. Analyze likely causes and recommend changes to sensing, estimation, data collection, and validation.

Multiple Choice Questions with Answers

Question 1

Which statement best describes the prediction step of a Bayesian filter?

A. It uses the motion model and control input to estimate the prior state distribution.

B. It permanently removes all measurement uncertainty.

C. It assigns every sensor return to a unique landmark.

D. It converts an occupancy grid into a topological map.

Correct Answer: A. It uses the motion model and control input to estimate the prior state distribution.

Question 2

The standard Kalman filter assumes which combination of conditions?

A. Nonlinear models and uniform noise

B. Linear models with additive Gaussian noise

C. A featureless environment and no controls

D. Only discrete-valued robot states

Correct Answer: B. Linear models with additive Gaussian noise

Question 3

In point-cloud registration, ICP primarily seeks to estimate:

A. A colour histogram

B. A rigid transformation aligning corresponding point sets

C. The camera’s exposure time

D. The probability of semantic object classes

Correct Answer: B. A rigid transformation aligning corresponding point sets

Question 4

What is the main purpose of a loop closure in SLAM?

A. To increase motor speed

B. To detect a previously visited place and reduce accumulated drift

C. To delete all old map points

D. To replace sensor calibration

Correct Answer: B. To detect a previously visited place and reduce accumulated drift

Question 5

A particle filter is especially useful when the robot’s belief is:

A. Guaranteed linear and deterministic

B. Multimodal or significantly non-Gaussian

C. Known exactly from wheel encoders

D. Independent of all measurements

Correct Answer: B. Multimodal or significantly non-Gaussian

Question 6

In the pinhole camera model, increasing depth Z while holding X constant generally causes the projected horizontal coordinate contribution f_x X/Z to:

A. Increase without limit

B. Decrease in magnitude

C. Remain exactly constant

D. Become a rotation angle

Correct Answer: B. Decrease in magnitude

Question 7

What does the covariance matrix primarily represent in a state estimator?

A. The robot’s maximum motor torque

B. Uncertainty and correlations among estimated variables

C. The number of map landmarks only

D. The image colour palette

Correct Answer: B. Uncertainty and correlations among estimated variables

Question 8

Why are dynamic pedestrians commonly excluded from a permanent structural map?

A. They cannot produce sensor measurements

B. Their changing positions would corrupt the representation of static structure

C. They always have zero velocity

D. They improve loop-closure accuracy automatically

Correct Answer: B. Their changing positions would corrupt the representation of static structure

Question 9

For the worked scalar fusion example, the updated position estimate was:

A. 8.4 m

B. 10.4 m

C. 11.6 m

D. 14.0 m

Correct Answer: C. 11.6 m

Question 10

Which factor is most directly associated with a false loop closure?

A. A geometrically verified unique landmark

B. An incorrect association between a current view and a visually similar place

C. A correctly synchronized timestamp

D. A calibrated sensor transformation

Correct Answer: B. An incorrect association between a current view and a visually similar place