Learning Control and Reinforcement Learning

Introduction

Intelligent control systems are increasingly required to operate in environments that are uncertain, changing, and difficult to model accurately. A conventional feedback controller depends on a fixed mathematical model or on parameters tuned by an engineer. Learning control supplements feedback with mechanisms that improve performance from repeated experience or from interaction with an environment.

Two related but distinct ideas are central. Iterative learning control (ILC) improves a control input when a task is repeated, such as robot welding or pick-and-place motion. Reinforcement learning (RL) learns a decision-making policy through trial and error, usually when actions influence both immediate and future outcomes. Both approaches are important in mechatronics, but both must be designed with explicit attention to stability, safety, data efficiency, and transfer from simulation to physical hardware.

Learning objectives

After studying this chapter, the reader should be able to:

1. Explain iterative learning control and distinguish it from adaptive and reinforcement learning.
2. Define states, actions, rewards, value functions, policies, and returns.
3. Derive and interpret the Bellman and temporal-difference learning equations.
4. Compare value-based, policy-based, and actor–critic methods.
5. Design a safe simulation-based learning controller for a mechatronic plant.
6. Evaluate exploration, sample efficiency, reward design, and sim-to-real transfer.
7. Identify governance, safety, and ethical requirements for learning controllers.

1. Iterative learning control

1.1 Basic idea

Suppose a plant performs the same finite-duration task during trials indexed by k. Let r(t) be the desired trajectory, y_k(t) the measured output, and u_k(t) the input applied during trial k. The tracking error is

e_k(t) = r(t) − y_k(t)

An ILC algorithm modifies the next trial’s input using the error from the previous trial:

Here, L is the learning operator or learning gain. In the simplest proportional form, L is a scalar α, so u_k+1(t) = u_k(t) + αe_k(t). The units of α must make αe_k have the same units as u. For example, if e is measured in metres and u is measured in newtons, α has units N/m.

ILC assumes that the task is repeated, the initial conditions are sufficiently repeatable, and the plant is reasonably consistent from one trial to the next. It is therefore not a general solution for one-time decisions or rapidly changing environments.

1.2 Learning and stability

For a linear time-invariant plant with input-output operator G and learning operator L, a simplified trial-to-trial error relation is

where I is the identity operator. Convergence requires the error-propagation operator to contract. In norm form, a sufficient condition is

The exact condition depends on the plant, signal norm, filtering, disturbances, and whether the system is continuous or discrete. A large learning gain may accelerate initial improvement but cause oscillation or divergence, especially when the plant has uncertain phase or unmodelled dynamics.

A practical update often includes a robustness filter Q:

Q attenuates high-frequency components that are poorly modelled or dominated by measurement noise. A feedback controller remains active during each trial; ILC should normally be treated as an additional feedforward-learning component rather than as a replacement for stabilizing feedback.

1.3 ILC compared with adaptive and reinforcement learning

ILC is especially effective for repetitive manufacturing. Adaptive control is suitable when plant parameters vary. RL is more general but usually requires substantially more data and stronger safety mechanisms.

2. Reinforcement-learning formulation

2.1 Markov decision processes

A common mathematical model for RL is the Markov decision process (MDP), defined by states S, actions A, transition probabilities, a reward function, and a discount factor γ. At time t, the system observes state s_t, selects action a_t, receives reward r_{t+1}, and enters state s_{t+1}.

The Markov property means that the distribution of the next state depends on the current state and action, not on the complete history:

P(s_t+1 | s₀,a₀,…,s_t,a_t) = P(s_t+1 | s_t,a_t)

This is an abstraction. In a real robot, unmeasured friction, payload, and contact conditions may make the observable state incomplete. Recurrent models, state estimation, or belief states may then be required.

The discount factor satisfies 0 ≤ γ < 1 for an infinite-horizon discounted problem. The return from time t is

Equivalently, G_t = r_t+1 + γG_t+1. Rewards must be dimensionally and operationally meaningful. They may represent a weighted engineering objective, but arbitrary scaling can affect learning algorithms, especially when entropy or regularization terms are included.

2.2 Policies and value functions

A policy π specifies how actions are selected. For a stochastic policy, π(a|s) = P(a_t = a | s_t = s). A deterministic policy is written a = μ(s).

The state-value function under policy π is

The action-value function is

The expectation accounts for stochastic transitions, rewards, and actions. The optimal value functions are

and

For a finite action set, an optimal greedy action is a* = arg max_a Q*(s,a). Value functions predict long-term performance; they do not themselves specify safe operation unless safety is incorporated into the state, reward, constraints, or supervisory controller.

2.3 Bellman equations

The Bellman expectation equation expresses a value recursively:

For continuous spaces, sums are replaced by integrals and transition probabilities by probability densities. The optimality equation is

These equations are foundational because they convert a long-horizon problem into a one-step relationship plus a continuation value.

3. Temporal-difference learning

3.1 TD prediction

Monte Carlo prediction waits until an episode ends to calculate the return. Temporal-difference (TD) learning updates estimates after each transition. The one-step TD target is

and the TD error is

For a tabular state-value estimate, the update is

where α is a dimensionless learning rate satisfying commonly 0 < α ≤ 1. At a terminal state, the continuation term is set to zero. TD methods are bootstrapping methods because they learn partly from an existing estimate.

3.2 Q-learning and SARSA

For action values, Q-learning uses an off-policy target:

It learns toward the greedy target even if the behaviour policy explores. SARSA is on-policy:

SARSA evaluates the policy that actually selects the next action. Consequently, with an exploratory policy, it may learn a more conservative behaviour than Q-learning. In continuous-action control, direct tabular methods become impractical; function approximation and algorithms such as DQN, DDPG, TD3, or SAC are used, each with assumptions and failure modes.

3.3 Worked numerical example

Assume γ = 0.9, learning rate α = 0.2, current estimate Q(s_t,a_t) = 4, reward r_t+1 = 2, and the largest next-state estimate is 5. The Q-learning target is

2 + 0.9×5 = 6.5.

Thus δ_t = 6.5 − 4 = 2.5, and the updated value is 4 + 0.2×2.5 = 4.5. This calculation assumes a nonterminal next state and a discrete action set for which the maximum is available.

4. Policy-gradient and actor–critic control

Value-based methods can be awkward when actions are continuous, as in motor voltage or torque control. Policy-gradient methods represent a differentiable policy π_θ with parameters θ and optimize expected return J(θ). A policy-gradient identity is

This form assumes suitable regularity conditions and an unbiased or controlled-variance estimate of the return. In practice, a baseline b(s_t) can reduce variance without changing the expected gradient:

An actor–critic architecture combines two learners. The actor represents the policy and selects actions; the critic estimates V, Q, or an advantage function. A common advantage estimate is

A(s_t,a_t) = Q(s_t,a_t) − V(s_t).

The critic’s TD error can provide an online advantage estimate:

where w denotes critic parameters. A conceptual actor update is

where β is the actor learning rate. The critic is updated separately, often by minimizing a squared TD error. Modern continuous-control algorithms add target networks, replay buffers, entropy regularization, or twin critics to improve numerical stability. These devices improve learning behaviour but do not guarantee physical safety.

5. Exploration, efficiency, and reward design

5.1 Exploration

Exploration gathers information about actions whose consequences are uncertain. Common strategies include ε-greedy selection, Gaussian action noise, entropy regularization, parameter noise, and intrinsically motivated rewards. Exploration must be bounded in a physical system. A robot should not discover a collision by executing an unbounded torque command.

Exploration can be separated into:

• Aleatoric uncertainty, inherent randomness in the plant or environment.
• Epistemic uncertainty, lack of knowledge that could be reduced with data.

A safe controller can restrict actions to a verified set, use a control barrier function or model-predictive safety filter, and allow a conventional fallback controller to override the learner.

5.2 Sample efficiency

Real experiments are expensive because they consume time, energy, and hardware life. Sample efficiency can be improved through model-based planning, demonstrations, offline datasets, prioritized replay, curriculum learning, representation learning, and transfer from related tasks. However, replaying correlated or biased data can produce misleading confidence. Dataset coverage and distribution shift must be measured, not assumed.

5.3 Reward design

A reward may combine tracking, effort, smoothness, and safety:

Here e_t is tracking error, u_t is control input, Δu_t is input change, the w terms are nonnegative weights, and P_t is a penalty for constraint violation. The terms must be normalized before weighting when their numerical scales differ. The expression is a design objective, not a physical law.

Reward design creates risks. A controller rewarded only for speed may overshoot; one rewarded only for low energy may fail to track; a large collision penalty may be ineffective if collisions are rare. Reward hacking occurs when the agent maximizes the specified reward while violating the designer’s actual intention. Safety constraints should therefore not rely solely on penalties. Hard action limits, termination rules, monitors, and independent safety controllers are preferable.

Potential-based reward shaping can preserve optimal policies under certain conditions. If a shaping term has the form

where Φ is a potential function, adding F to the reward preserves optimal policies in the standard discounted MDP under the relevant assumptions. Arbitrary shaping does not have this guarantee.

6. Safe simulation-based controller design

A practical workflow is:

1. Define the task, state variables, actions, constraints, and termination conditions.
2. Build a simulator containing rigid-body dynamics, actuators, sensors, delays, friction, contact, and disturbances.
3. Validate the simulator against measured data and quantify model uncertainty.
4. Train initially with action limits and a stabilizing baseline controller.
5. Apply safety filters, constraint monitors, and emergency stop logic.
6. Evaluate across randomized masses, friction coefficients, delays, sensor noise, and disturbances.
7. Compare the learned controller with a tuned baseline using identical test conditions.
8. Transfer gradually through hardware-in-the-loop, low-energy tests, and supervised operating envelopes.
9. Log actions, observations, safety interventions, and software versions.
10. Retest after every policy or plant change.

A useful architecture is hierarchical. A high-level policy selects references or controller parameters, while a verified low-level feedback controller tracks them. This reduces the action space and prevents the learning algorithm from directly commanding unsafe motor currents.

7. Sim-to-real transfer

A policy trained in simulation may fail on hardware because of the reality gap: inaccurate dynamics, unmodelled compliance, actuator saturation, latency, quantization, sensor bias, or environmental differences. Important transfer techniques include:

• System identification: estimate mass, inertia, friction, motor constants, and delays from experiments.
• Domain randomization: vary uncertain parameters during training so the policy learns robust behaviour.
• Residual learning: retain a trusted model-based controller and learn only a correction.
• Progressive deployment: increase speed, force, and workspace only after passing safety tests.
• Calibration and normalization: ensure simulated and physical observations have the same units and ranges.
• Uncertainty monitoring: detect observations outside the training distribution.

Domain randomization is not automatically beneficial. If ranges are unrealistic, learning may become inefficient; if critical real-world effects are omitted, randomization cannot correct the gap. Transfer should be treated as an engineering validation problem rather than a final software setting.

8. Illustrative case: learning control for a two-axis pick-and-place robot

Background

Consider a laboratory two-axis robot that moves a gripper between fixed pickup and placement locations. A conventional computed-torque controller provides stable motion, but payload changes produce overshoot and longer settling times.

Problem

Direct RL torque control is considered unsafe because excessive motor commands could damage the gearbox. The task is repetitive but payload and friction vary.

Analysis

The repetitive trajectory makes ILC suitable for reducing systematic tracking error. RL is useful for selecting trajectory speed and feedforward corrections under different payload estimates. The state includes joint positions, velocities, estimated payload, tracking error, and previous command. The action is a bounded correction to the reference velocity and feedforward torque. A low-level feedback controller remains responsible for stabilization.

Approach

First, the plant is identified from low-speed experiments. A digital-twin simulator randomizes payload, friction, sensor noise, and delay. The actor–critic policy is trained with a reward penalizing position error, energy, command changes, and constraint violations. Episodes terminate when joint limits, excessive current, or workspace boundaries are approached. A supervisory safety filter clips commands and switches to the baseline controller when predicted constraints are violated.

During physical deployment, the system proceeds through four stages: hardware-in-the-loop simulation, unloaded low-speed motion, moderate payload tests, and normal production speed. ILC updates the repeated feedforward trajectory only after a complete trial and filters high-frequency corrections.

Outcome

The expected engineering outcome is reduced repeatable tracking error without sacrificing the baseline controller’s stability. Performance must be reported using independently collected test trials, not only training returns. Relevant measures include root-mean-square position error,

where N is the number of sampled errors, peak overshoot, settling time, energy consumption, number of safety interventions, and successful cycle percentage. A learned policy that improves reward but increases emergency stops is not an acceptable improvement.

Learning points

The case illustrates that ILC exploits repetition, RL handles changing decision conditions, and a layered architecture limits risk. It also shows why simulator validation, conservative action spaces, independent safety logic, and evaluation on unseen conditions are essential.

9. Limitations, governance, and professional practice

Learning controllers may be unstable during training, sensitive to random seeds, difficult to verify, and opaque to operators. Function approximation can cause extrapolation errors, especially when a policy encounters states absent from its training data. Safety claims must distinguish simulation evidence from physical evidence.

Professional deployment should include hazard analysis, traceable requirements, access control, audit logs, versioned models, reproducible training configurations, and documented rollback procedures. Human operators need clear indications of when the learned controller is active and when the safety supervisor has intervened. Data collected from workplaces may contain personal or proprietary information; collection and retention must follow applicable privacy and cybersecurity rules.

For safety-related machinery, learning components should be isolated from independently verified protective functions wherever possible. Ethical design requires attention to operator safety, accountability, foreseeable misuse, energy consumption, and unequal effects on workers. An autonomous decision is not exempt from engineering responsibility merely because it was produced by a learned policy.

10. Contemporary directions

Current research combines RL with model-predictive control, differentiable simulation, offline reinforcement learning, uncertainty-aware policies, and foundation models for task specification. Offline RL can reduce dangerous exploration by learning from previously collected data, but it is vulnerable to distributional extrapolation. Safe RL incorporates constraints directly through constrained optimization, shielding, reachability analysis, or risk-sensitive objectives. These methods remain active research areas; claims of guaranteed safety require explicit assumptions about models, disturbances, sensing, and computation.

Chapter Summary

Iterative learning control improves repeated tasks by updating future inputs from previous tracking errors. Its convergence depends on the plant, learning operator, filtering, and repeatability. Reinforcement learning models sequential interaction using states, actions, rewards, transitions, policies, returns, and value functions. Bellman equations express long-term value recursively. TD learning updates estimates from one-step prediction errors; Q-learning is off-policy, whereas SARSA is on-policy. Policy-gradient methods optimize parameterized policies, and actor–critic methods combine policy learning with value estimation.

Effective mechatronic learning controllers require bounded exploration, carefully scaled rewards, sample-efficient training, validated simulation, domain randomization, and progressive sim-to-real testing. A layered design—learned high-level decisions, conventional low-level control, and independent safety supervision—is generally more defensible than unrestricted end-to-end torque learning. Evaluation must include tracking, energy, robustness, constraint violations, and safety interventions, not reward alone.

Key Terms

Iterative learning control: A method that improves a repeated task using errors from earlier trials.

Learning operator: The mapping that converts a previous trial’s error into an input correction.

Markov decision process: A sequential decision model defined by states, actions, transitions, rewards, and discounting.

Policy: A rule or probability distribution for selecting actions from states.

Return: The discounted sum of present and future rewards.

Value function: Expected return from a state under a specified policy.

Action-value function: Expected return from a state-action pair.

Bellman equation: A recursive relationship between current value and successor value.

Temporal-difference error: Difference between a one-step target and the current value estimate.

Bootstrapping: Updating an estimate using another learned estimate.

Q-learning: An off-policy TD method that uses the greedy next-action value.

SARSA: An on-policy TD method named for state, action, reward, next state, and next action.

Policy gradient: A method that adjusts policy parameters using the gradient of expected return.

Actor–critic: An architecture with a policy learner and a value-estimating critic.

Exploration: Selection of uncertain actions to obtain information.

Sample efficiency: The performance gained per unit of collected experience.

Reward shaping: Modification of rewards to guide learning toward desired behaviour.

Reality gap: Difference between simulated and physical system behaviour.

Domain randomization: Training across varied simulated parameters to improve robustness.

Safety filter: An independent mechanism that restricts or overrides unsafe actions.

Review Questions

Short Answer Questions

1. Why is repeatability an important assumption in iterative learning control?
2. Define a policy and a value function in reinforcement learning.
3. What is the difference between on-policy and off-policy learning?
4. Why can reward penalties alone be insufficient for physical safety?
5. Name two causes of the simulation-to-reality gap.

Long Answer Questions

1. Derive and explain the one-step TD update for a state-value function.
2. Compare ILC, adaptive control, and RL in terms of assumptions, learning signals, and applications.
3. Explain the roles of the actor and critic in an actor–critic controller.
4. Discuss how exploration and sample efficiency should be managed in a robotic system.
5. Describe a complete workflow for transferring a learned controller from simulation to hardware.

Analytical / Application Questions

1. A robot repeatedly follows a trajectory but develops high-frequency oscillations after each ILC update. Analyze possible causes and propose corrective design changes.
2. Design a reward and safety architecture for a mobile manipulator operating near humans. Identify which requirements should be hard constraints rather than reward terms.
3. A policy performs well in simulation but causes overshoot on hardware. Develop a diagnostic plan that separates model error, delay, sensor mismatch, action scaling, and policy extrapolation.

Multiple Choice Questions with Answers

Question 1

Which condition is most directly associated with successful iterative learning control?

A. The task is repeated with sufficiently consistent initial conditions

B. The action space must contain only two actions

C. The reward must be sparse

D. The plant must have no feedback controller

Correct Answer: A. The task is repeated with sufficiently consistent initial conditions

Question 2

What does the TD error δ_t = r_t+1 + γV(s_t+1) − V(s_t) measure?

A. The actuator’s electrical power

B. The difference between a one-step target and the current value estimate

C. The probability of selecting an action

D. The simulation time step

Correct Answer: B. The difference between a one-step target and the current value estimate

Question 3

Which method is explicitly off-policy in its standard tabular form?

A. SARSA

B. Monte Carlo policy evaluation

C. Q-learning

D. On-policy actor updating

Correct Answer: C. Q-learning

Question 4

In an actor–critic architecture, the critic primarily:

A. Restricts motor current mechanically

B. Estimates value information used to improve the policy

C. Randomizes all simulator parameters

D. Replaces the plant sensors

Correct Answer: B. Estimates value information used to improve the policy

Question 5

If γ = 0.9, the immediate reward is 3, and the next-state value is 10, what is the one-step TD target for a nonterminal transition?

A. 3.9

B. 10.0

C. 12.0

D. 13.0

Correct Answer: D. 13.0

Question 6

Which practice most directly limits unsafe exploration on a physical robot?

A. Removing all action limits

B. Using an independent safety filter and bounded action set

C. Increasing reward variance

D. Training only on terminal rewards

Correct Answer: B. Using an independent safety filter and bounded action set

Question 7

What is a major risk of arbitrary reward shaping?

A. It always eliminates exploration

B. It can encourage behaviour that maximizes the numerical reward but violates the real objective

C. It makes all policies optimal

D. It prevents simulation entirely

Correct Answer: B. It can encourage behaviour that maximizes the numerical reward but violates the real objective

Question 8

Which technique deliberately varies simulated parameters such as friction and payload during training?

A. Domain randomization

B. Greedy evaluation

C. Terminal bootstrapping

D. Action discretization

Correct Answer: A. Domain randomization

Question 9

For N sampled tracking errors, which expression is the root-mean-square error?

A. Σe_i/N

B. √(Σe_i²/N)

C. N/Σe_i²

D. Σ|e_i|²

Correct Answer: B. √(Σe_i²/N)

Question 10

Why is a hierarchical learning architecture often preferred for safety-critical mechatronics?

A. It allows the policy to bypass feedback control

B. It confines learning to higher-level decisions while verified low-level control maintains stability

C. It guarantees perfect sim-to-real transfer

D. It removes the need for testing

Correct Answer: B. It confines learning to higher-level decisions while verified low-level control maintains stability