Optimization, Reinforcement Learning and Decision-Making

Introduction

Mechatronic systems operate under competing objectives, physical constraints, incomplete information and changing environments. A mobile robot may need to minimize energy while reaching a destination quickly; a production cell may maximize throughput while limiting vibration and wear; an autonomous vehicle may choose a maneuver that is safe even when sensor measurements are uncertain. These are decision problems rather than simple control problems.

Optimization provides systematic methods for selecting the best feasible design or action. Bayesian methods provide a principled way to represent and update uncertainty. Markov decision processes (MDPs) model sequential decisions, while reinforcement learning (RL) learns policies through interaction with an environment. In practical engineering, these methods must be combined with physical models, safety constraints, validation procedures and human oversight.

This chapter develops these ideas for mechatronics applications, with emphasis on heuristic optimization, Bayesian decision-making, MDPs, reward design, sample efficiency and safe exploration.

Learning objectives

By the end of this chapter, a student should be able to:

1. Formulate constrained engineering decisions mathematically.
2. Distinguish deterministic, stochastic, heuristic and learning-based optimization.
3. Apply genetic algorithms, particle swarm optimization and simulated annealing appropriately.
4. Explain Bayesian updating and quantify decision uncertainty.
5. Formulate a mechatronic task as an MDP.
6. Compare value-based, policy-based and model-based reinforcement learning.
7. Design rewards without creating unsafe or unintended behavior.
8. Select methods using computational cost, sample efficiency and safety requirements.

1. Engineering decisions as constrained optimization

A decision variable is a quantity selected by a designer or controller. Examples include motor speed, gear ratio, trajectory parameters, controller gains and inspection intervals. An objective function assigns a cost or utility to each decision. A general constrained optimization problem is

Here, x is the decision vector, J(x) is the scalar cost, g_i(x) are inequality constraints, h_j(x) are equality constraints and X is the permissible design domain. The formulation is applicable when the objectives and constraints can be evaluated for candidate decisions. Units must be consistent: a weighted cost should either use normalized quantities or explicitly stated conversion factors.

For multiple objectives, a common scalarization is

where J_k are normalized objective terms and w_k are preference weights. Without normalization, an objective measured in watts may dominate one measured in seconds simply because of numerical scale. A solution is Pareto-optimal if no objective can be improved without worsening at least one other objective.

Feasibility and uncertainty

A feasible solution satisfies every constraint. In engineering, constraints may include motor current, temperature, acceleration, workspace, collision clearance, battery state and structural stress. If a parameter is uncertain, a deterministic constraint may be inadequate. For example, requiring a nominal temperature below a limit does not guarantee safety when the temperature model is inaccurate.

A chance constraint expresses probabilistic safety:

where ξ is an uncertain quantity and α is the allowable violation probability. This formulation assumes that a probability model for ξ is credible and that repeated-operation interpretation is meaningful. A robust alternative requires feasibility for all uncertainties in a specified set, which is often more conservative.

2. Classical and heuristic optimization

Gradient-based methods are efficient when the objective is smooth, differentiable and reasonably well behaved. Mechatronic objectives are frequently discontinuous or expensive: a simulator may fail for an unstable design, a collision checker may return a binary result, or a hardware experiment may be noisy. Heuristic methods are useful in such cases because they search using objective evaluations rather than derivatives.

Genetic algorithms

A genetic algorithm (GA) represents candidate solutions as chromosomes. A population is evaluated, better candidates are selected, and crossover and mutation generate new candidates. A typical cycle is:

1. Generate an initial population, preferably including physically informed designs.
2. Evaluate cost and constraint violations.
3. Select candidates using fitness or ranking.
4. Apply crossover and mutation.
5. Preserve a small elite set if loss of the best solution is undesirable.
6. Repeat until a stopping criterion is reached.

Constraint handling may use a penalty function such as

where λ is a penalty coefficient. This is not equivalent to enforcing constraints exactly: a high penalty can make the search numerically difficult, while a low penalty can favor infeasible solutions. Repair operators or feasibility-preserving encodings are often preferable for safety-critical design.

Particle swarm optimization

In particle swarm optimization (PSO), particle i has position x_i and velocity v_i. A common update is

Here, p_i is the particle’s best position, g is the swarm’s best position, ω is an inertia parameter, c₁ and c₂ are acceleration coefficients, and r₁,r₂ are independent random numbers uniformly distributed between 0 and 1. Positions and velocities must be bounded when physical variables have limits. PSO can converge rapidly but may prematurely lose diversity.

Simulated annealing

Simulated annealing accepts some worse solutions to escape local minima. If a candidate changes cost by ∆J > 0, it is accepted with probability

where T > 0 is a temperature parameter. As temperature decreases, exploration reduces. The expression assumes costs are comparable and dimensionless after scaling; otherwise T must carry the same units as cost. Cooling too quickly causes local trapping, while cooling too slowly is computationally expensive.

Worked example: a simple constrained design

Suppose a gripper designer selects jaw speed v in metres per second. The estimated cost is

subject to 0.1 ≤ v ≤ 0.8. Differentiating gives dJ/dv = 2(v − 0.4) + 0.1. Setting this to zero yields v = 0.35 m/s, which lies within the feasible interval. The cost is J(0.35) = (−0.05)² + 0.035 = 0.0375. Since the objective is convex in this interval, this stationary point is the global minimum. A heuristic algorithm could also find it, but would be unnecessary for this smooth one-dimensional problem. This illustrates a central principle: use the simplest reliable method that matches the problem structure.

3. Bayesian reasoning for uncertain engineering decisions

Bayesian inference updates a prior belief using evidence. For a hypothesis or parameter θ and data D,

The posterior p(θ|D) combines the likelihood p(D|θ) with the prior p(θ). The denominator p(D) is the evidence and ensures normalization. Bayesian reasoning is especially useful when experiments are costly and prior physical knowledge exists.

For a binary fault hypothesis F, the posterior odds form is

The first ratio is the likelihood ratio and the second is the prior odds. This equation shows why a rare fault may remain unlikely after weak evidence, but become highly probable after a diagnostic signal strongly associated with failure.

Numerical Bayesian example

Assume a bearing fault has prior probability 0.02. A vibration alarm detects 90% of faulty bearings and falsely alarms on 5% of healthy bearings. Then

The numerator is 0.018 and the denominator is 0.067, giving approximately P(F|A) = 0.269. Thus, an alarm raises the probability from 2% to about 26.9%, but does not prove failure. A maintenance decision should also consider inspection cost, downtime and the consequences of missed failure.

Bayesian optimization

Bayesian optimization is appropriate when evaluating J(x) is expensive. A surrogate model, often a Gaussian process, estimates both predicted performance and uncertainty. An acquisition function selects the next experiment by balancing exploitation of promising regions and exploration of uncertain regions. Expected improvement, for example, rewards candidates likely to improve the current best result.

Its advantages include sample efficiency and uncertainty estimates. Its limitations include difficulty with very high-dimensional spaces, poor surrogate quality under discontinuities and the need to account explicitly for constraints and noisy measurements. In a hardware setting, every proposed experiment should pass a safety filter before execution.

4. Markov decision processes

An MDP models sequential decision-making using the tuple (S,A,P,R,γ). S is the state space, A the action space, P(s'|s,a) the transition probability, R(s,a,s') the reward and γ ∈ [0,1) the discount factor. The Markov property requires that the next-state distribution depends on the past only through the current state and action:

A state must contain sufficient information for decision-making. If a robot’s position is known but its velocity is omitted, the process may appear non-Markovian because future motion depends on velocity. Partial observability can be addressed with a belief state, recurrent model or state estimator.

The return from time t is

where r_t+k+1 is the reward received after each transition. The value of a policy π is

and the action-value function is

The Bellman expectation equation is

For continuous or stochastic systems, sums may be replaced by integrals. The equation assumes a well-defined reward and discounting scheme. Discounting emphasizes near-term performance; it is not automatically a physical measure of energy or risk.

5. Reinforcement learning methods

RL seeks a policy that maximizes expected cumulative reward through interaction or data. In model-free learning, the transition model is not explicitly known. In model-based RL, a model is learned or supplied and used for planning.

Q-learning

For discrete actions, Q-learning uses the update

where α is the learning rate. The bracketed term is the temporal-difference error. Under standard theoretical conditions, including sufficient exploration and suitable diminishing learning rates, tabular Q-learning can converge to an optimal action-value function in finite MDPs. Function approximation, continuous states and nonstationary data weaken these guarantees.

Policy gradients and actor–critic methods

Policy-gradient methods optimize parameters θ of a stochastic policy π_θ. A general objective is

where τ denotes a trajectory. Actor–critic methods use an actor to select actions and a critic to estimate value. They are suitable for continuous motor commands but can be sensitive to reward scaling, exploration noise and unstable training.

Sample efficiency

Real robots provide few, expensive and sometimes unsafe trials. Sample efficiency can be improved by:

• learning from demonstrations;
• using simulators and domain randomization;
• incorporating analytical dynamics or a digital twin;
• using replay buffers and off-policy data;
• selecting informative experiments;
• transferring policies from simpler tasks; and
• terminating unsafe episodes early.

Simulation-to-reality transfer is limited by model mismatch. Randomizing mass, friction, latency, sensor noise and actuator response can improve robustness, but excessive randomization may make learning inefficient.

6. Reward design and safe exploration

A reward is not merely a performance score; it defines what the agent is incentivized to do. A mobile robot reward might combine progress, energy, time and safety:

where ∆d_t is progress, E_t is energy consumption, ∆t is elapsed time and C_t represents collision or constraint violation. The weights must be calibrated and terms normalized. A large progress reward can unintentionally encourage unsafe shortcuts.

Reward hacking occurs when an agent maximizes the specified reward while violating the designer’s real intention. Examples include repeatedly triggering a sensor that supplies a reward, stopping before a difficult final maneuver, or exploiting simulator errors. Safety-critical requirements should not rely solely on penalties. Hard action limits, barrier functions, runtime monitors and independent emergency stops are stronger safeguards.

A constrained RL formulation maximizes expected return subject to an expected cost limit:

where c_t is a safety cost and d is an acceptable bound. This formulation is meaningful only if the cost captures the hazard and the bound is justified. Expected constraints alone may still permit rare catastrophic events, so worst-case or chance constraints may be necessary.

Safe exploration strategies include action shielding, conservative uncertainty bounds, safe sets, constrained policy updates, human approval and offline pretraining. Exploration noise should never be allowed to violate actuator, thermal, collision or stability limits.

7. Integrated illustrative case: autonomous warehouse vehicle

Background

The following is an illustrative case. A battery-powered autonomous vehicle transports bins between storage and assembly stations. It has wheel encoders, lidar, inertial sensing and motor-current measurements. The vehicle must minimize delivery time and energy while avoiding people, obstacles and low-battery operation.

Problem formulation

A state may include position, velocity, local obstacle distances, battery state, load estimate and a short history of sensor-quality indicators. The action is a pair of commanded linear and angular velocities, bounded by actuator and stability limits. The transition model includes uncertain wheel slip, latency and obstacle motion.

A purely shortest-path planner performs well in empty aisles but becomes inefficient under congestion. A learned policy could adapt speed and route selection, but unsafe exploration near workers is unacceptable.

Approach

First, a model-based planner generates collision-free trajectories using conservative obstacle inflation. Demonstration data from this planner and trained operators initialize the policy. A simulator randomizes friction, sensor noise, payload and communication delay. The policy reward includes progress and energy terms, while collision, minimum-clearance and excessive acceleration are handled by hard action shielding and episode termination.

A Bayesian model estimates uncertainty in predicted travel time and battery consumption. When uncertainty is high, the decision layer selects a slower route or requests a new observation rather than blindly exploiting the nominally fastest route. During deployment, a runtime monitor checks speed, clearance, battery temperature and sensor validity. Any violation triggers a verified fallback controller.

Outcome and evaluation

The policy is evaluated in simulation, then in a test zone, then during supervised operation. Important metrics are successful deliveries, collision and near-miss rates, energy per delivery, intervention frequency, computation time and performance under payload and sensor changes. Average reward alone is insufficient because it can hide rare but severe failures.

Learning points

The case demonstrates that optimization selects designs and routes, Bayesian inference manages uncertainty, MDPs represent sequential decisions, RL adapts behavior, and safety mechanisms constrain what learning may do. No single algorithm replaces system engineering, validation or operational governance.

8. Professional practice, governance and limitations

Engineers should document the objective, constraints, data sources, simulator assumptions, training distribution and known failure modes. Reproducibility requires recorded software versions, random seeds where relevant, sensor calibration and evaluation protocols. Independent test data are essential; repeatedly tuning on the test set invalidates performance claims.

Safety standards and organizational procedures may require hazard analysis, traceability, verification and validation, access control and incident reporting. For systems affecting workers or the public, accountability must remain with responsible engineers and operators. Data used for learning should respect privacy and cybersecurity requirements. A learned policy should be auditable enough to support diagnosis, even when its internal representation is complex.

Important limitations include the curse of dimensionality, distribution shift, sim-to-real mismatch, reward misspecification, unstable function approximation and rare-event risk. Heuristic optimization may provide no proof of global optimality. Bayesian estimates are only as credible as their prior and likelihood. RL performance can be highly sensitive to random seeds and environment design. These limitations make layered architectures—planning, learning, monitoring and fallback control—preferable to unconstrained end-to-end autonomy in many mechatronic systems.

9. Method-selection guide

Chapter Summary

Engineering decision-making combines objectives, constraints and uncertainty. Optimization formulates feasible choices, while heuristic methods are useful when objectives are discontinuous, noisy or expensive. Genetic algorithms use population evolution, PSO uses social search and simulated annealing uses controlled acceptance of worse candidates.

Bayesian inference updates prior knowledge with evidence and supports uncertainty-aware diagnosis and experiment selection. MDPs formalize sequential decisions through states, actions, transitions, rewards and discounting. Reinforcement learning estimates values or policies from interaction, but its usefulness depends strongly on state representation, reward design, sample efficiency and safe exploration.

For mechatronic systems, safety should be implemented through constraints, shielding, monitoring, conservative deployment and verified fallback controllers—not through reward penalties alone. Evaluation must include reliability, rare failures, energy, computation and human intervention, rather than average reward only.

Key Terms

Decision variable: A quantity selected by a designer or controller.

Objective function: A function measuring cost, performance or utility.

Constraint: A condition that a feasible solution must satisfy.

Pareto-optimality: A condition in which no objective can improve without worsening another.

Chance constraint: A probabilistic requirement on constraint satisfaction.

Genetic algorithm: A population-based search method using selection, crossover and mutation.

Particle swarm optimization: A population method in which particles update using personal and collective best positions.

Simulated annealing: A stochastic search method that sometimes accepts worse solutions to escape local minima.

Bayesian inference: Updating a prior distribution using observed evidence.

Likelihood: The probability of observed data conditional on a hypothesis or parameter.

Markov decision process: A model of sequential decisions satisfying the Markov property.

Policy: A rule assigning actions or action probabilities to states.

Return: Discounted cumulative future reward.

Value function: Expected return from a state under a policy.

Q-function: Expected return from a state–action pair.

Temporal-difference error: Difference between a current estimate and a reward-based target estimate.

Sample efficiency: Performance achieved per unit of interaction data.

Reward hacking: Obtaining high specified reward while violating the intended objective.

Safe exploration: Learning or experimentation subject to safety constraints.

Simulation-to-reality transfer: Applying behavior learned in simulation to physical hardware.

Review Questions

Short Answer Questions

1. What distinguishes a constraint from an objective function?
2. Why should objective terms commonly be normalized before weighted addition?
3. State the Markov property in words.
4. What is the purpose of a runtime safety shield in reinforcement learning?
5. Why can an alarm with high sensitivity still produce many false alarms?

Long Answer Questions

1. Compare genetic algorithms, particle swarm optimization and simulated annealing for optimizing a mechatronic design with discontinuous simulation outcomes.
2. Explain Bayesian updating using the bearing-diagnosis example and discuss how maintenance cost could affect the final decision.
3. Formulate an autonomous robot task as an MDP, defining its state, action, transition, reward and discount factor.
4. Explain how reward misspecification and reward hacking can arise in a mobile-robot application.
5. Discuss a complete strategy for improving RL sample efficiency while controlling sim-to-real and safety risks.

Analytical / Application Questions

1. A robot may choose between a fast route with uncertain obstacle motion and a slower route with reliable clearance. Propose a Bayesian or chance-constrained decision formulation and identify the required data.
2. Design a constrained reward and independent safety architecture for a battery-powered manipulator operating near a human worker. Justify which requirements should be hard constraints rather than penalties.
3. Select an optimization or learning method for tuning a controller whose simulation takes ten minutes per trial, has five continuous parameters and includes noisy stability failures. Defend the choice and propose an evaluation protocol.

Multiple Choice Questions with Answers

Question 1

Which statement best describes a feasible solution to a constrained optimization problem?

A. It maximizes every objective independently.

B. It satisfies all specified constraints.

C. It has the largest numerical objective value.

D. It requires no model of the system.

Correct Answer: B. It satisfies all specified constraints.

Question 2

In the PSO update, what does the vector g commonly represent?

A. The particle's current acceleration

B. The gradient of the objective

C. The swarm's best-known position

D. A random mutation vector

Correct Answer: C. The swarm's best-known position

Question 3

A bearing fault has prior probability 0.02, alarm sensitivity 0.90 and false-alarm probability 0.05. What is approximately P(fault | alarm)?

A. 0.020

B. 0.050

C. 0.180

D. 0.269

Correct Answer: D. 0.269

Question 4

Which condition is required by the Markov property?

A. Actions must be continuous.

B. Rewards must always be positive.

C. The next-state distribution depends on the past through the current state and action.

D. The transition model must be deterministic.

Correct Answer: C. The next-state distribution depends on the past through the current state and action.

Question 5

What is the main purpose of the discount factor γ in an MDP?

A. To weight future rewards relative to immediate rewards.

B. To enforce actuator saturation.

C. To normalize sensor measurements.

D. To guarantee safe exploration.

Correct Answer: A. To weight future rewards relative to immediate rewards.

Question 6

In Q-learning, the temporal-difference error compares the current Q estimate with which quantity?

A. Only the previous action

B. The immediate reward plus discounted best next-state Q value

C. The average population fitness

D. The Bayesian prior probability

Correct Answer: B. The immediate reward plus discounted best next-state Q value

Question 7

Which is the clearest example of reward hacking?

A. An agent obeys a hard collision limit.

B. An agent uses less energy and reaches its goal.

C. An agent exploits a sensor loophole to obtain reward without completing the intended task.

D. An agent is trained using demonstrations.

Correct Answer: C. An agent exploits a sensor loophole to obtain reward without completing the intended task.

Question 8

Why is Bayesian optimization often suitable for hardware experiments?

A. It requires no objective evaluations.

B. It can select informative trials when evaluations are expensive.

C. It guarantees a global optimum after one trial.

D. It eliminates all measurement noise.

Correct Answer: B. It can select informative trials when evaluations are expensive.

Question 9

Which measure is most directly associated with sample efficiency?

A. Performance achieved per amount of interaction data

B. Number of reward terms in the objective

C. Number of actuators in the robot

D. Maximum motor voltage

Correct Answer: A. Performance achieved per amount of interaction data

Question 10

Why should safety-critical RL systems use mechanisms beyond reward penalties?

A. Penalties always prevent learning.

B. Penalties can be ignored or traded against other rewards, allowing unsafe behavior.

C. Hard constraints make simulation impossible.

D. Rewards cannot be computed numerically.

Correct Answer: B. Penalties can be ignored or traded against other rewards, allowing unsafe behavior.