Reinforcement Learning and Sequential Decision-Making

Introduction and Context

Many AI systems do not make a single prediction. They repeatedly observe a changing environment, select actions, receive feedback, and influence what happens next. A robot chooses movements, a recommender selects items, a data-center controller allocates computing capacity, and an inventory system decides when to reorder. These are sequential decision-making problems.

Reinforcement learning (RL) studies how an agent can learn effective behavior through interaction with an environment. Unlike supervised learning, the agent is not normally given a correct action for every situation. Instead, it receives rewards or penalties, often delayed, and must discover which actions produce desirable long-term outcomes. Unlike ordinary optimization, the decisions affect future states and therefore future opportunities.

This chapter develops the mathematical framework of Markov decision processes (MDPs), value-based and policy-based learning, exploration, simulation, offline RL, safety, and practical implementation with TensorFlow and PyTorch. The emphasis is on assumptions, design choices, evaluation, and responsible deployment.

Learning Objectives

After completing this chapter, a student should be able to:

1. Formulate a sequential problem as an MDP and explain the Markov property.

2. Derive and interpret return, value functions, Bellman equations, and temporal-difference errors.

3. Implement the main ideas of Q-learning, deep Q-learning, policy gradients, and actor–critic learning.

4. Compare exploration strategies and distinguish online, simulated, and offline learning.

5. Evaluate RL systems using performance, safety, robustness, and statistical criteria.

6. Analyze applications in control, recommendation, and resource allocation.

7. Identify ethical, operational, and governance risks in reinforcement-learning deployment.

1. Reinforcement Learning as a Decision Problem

An RL system consists of an agent and an environment. At time step t, the agent observes a state S_t, selects an action A_t according to a policy, receives reward R_{t+1}, and transitions to S_{t+1}. The policy, written as π(a|s), specifies the probability of selecting action a in state s. A deterministic policy selects one action; a stochastic policy assigns probabilities to several actions.

The reward is a scalar signal designed to express progress toward an objective. It is not necessarily the same as business value, human well-being, or the final evaluation metric. Reward design is therefore a modeling and governance decision, not merely a programming detail.

The agent seeks to maximize expected cumulative reward. For a continuing or infinite-horizon problem, the discounted return from time t is

G_t = R_{t+1} + γR_{t+2} + γ²R_{t+3} + ⋯ = Σ_{k=0}^{∞} γ^k R_{t+k+1},

where γ is the discount factor, 0 ≤ γ < 1. Discounting makes distant rewards less influential and ensures a finite return when rewards are bounded. In an episodic task, the sum ends at the terminal time T. γ close to 1 emphasizes long-term outcomes; a smaller γ emphasizes immediate outcomes. Discounting should not be used to hide serious delayed harms.

The state-value function under policy π is

V^π(s) = E_π[G_t | S_t = s],

and the action-value function is

Q^π(s,a) = E_π[G_t | S_t=s, A_t=a].

The optimal value functions are V(s) = max_π V^π(s) and Q(s,a) = max_π Q^π(s,a). An optimal policy can be greedy with respect to Q: π(s) ∈ argmax_a Q*(s,a).

2. Markov Decision Processes

A finite MDP is commonly represented by the tuple (S, A, P, R, γ):

• S is the set of states.
• A is the set of actions.
• P(s'|s,a) is the transition probability from s to s' after action a.
• R(s,a) or R(s,a,s') is the expected immediate reward.
• γ is the discount factor.

The Markov property states that the distribution of the next state and reward depends on the history only through the current state and action:

P(S_{t+1},R_{t+1}|S_0,A_0,...,S_t,A_t) = P(S_{t+1},R_{t+1}|S_t,A_t).

This does not mean the physical world is memoryless. It means that the chosen state representation contains sufficient information for prediction and decision-making. If an autonomous vehicle receives only a single image, hidden velocity or intent may violate the practical Markov assumption. Frame stacking, recurrent networks, or belief states may be needed.

The Bellman expectation equation decomposes a value into immediate reward and future value:

V^π(s) = Σ_a π(a|s) Σ_{s'} P(s'|s,a)[R(s,a,s') + γV^π(s')].

For action values,

Q^π(s,a) = Σ_{s'}P(s'|s,a)[R(s,a,s') + γΣ_{a'}π(a'|s')Q^π(s',a')].

The Bellman optimality equations are

V(s) = max_a Σ_{s'}P(s'|s,a)[R(s,a,s') + γV(s')],

Q(s,a) = Σ_{s'}P(s'|s,a)[R(s,a,s') + γ max_{a'}Q(s',a')].

These equations assume a stationary MDP and a compatible discounted objective. They are especially useful when the state and action spaces are small enough for dynamic programming. Value iteration repeatedly applies the optimality operator, while policy iteration alternates policy evaluation and policy improvement. Both require a known model and become impractical in large or unknown environments.

Worked Example: A Two-State MDP

Suppose a state A has two actions. Action stay gives reward 2 and returns to A. Action move gives reward 5 and transitions to terminal state B. Let γ = 0.9 and V(B)=0. If the agent always stays, then V(A)=2+0.9V(A), so 0.1V(A)=2 and V(A)=20. If it moves immediately, its value is 5. Thus, under these specified rewards, continuing to stay is optimal even though moving gives a larger immediate reward. The example illustrates why an RL objective must describe the intended long-term outcome and why immediate reward alone is an inadequate decision criterion.

3. Temporal-Difference Learning and Q-Learning

When the transition model is unknown, an agent can learn from sampled experience. Monte Carlo methods wait until an episode ends and use the observed return. Temporal-difference (TD) methods update using an estimate of the next state before the episode ends.

For a value function, the one-step TD target is

Y_t = R_{t+1} + γV(S_{t+1}),

and the TD error is

δ_t = Y_t - V(S_t).

A tabular update is V(S_t) ← V(S_t) + αδ_t, where α is a learning rate. For a terminal transition, the future term is zero.

Q-learning is an off-policy control algorithm. Its update is

Q(S_t,A_t) ← Q(S_t,A_t) + α[R_{t+1}+γ max_{a'}Q(S_{t+1},a')−Q(S_t,A_t)].

The target uses the greedy next action even if the behavior policy explores. Under standard conditions—finite MDP, sufficient exploration of state-action pairs, appropriately decreasing learning rates, and a stationary environment—tabular Q-learning converges to Q* with probability one. Those guarantees do not automatically apply to neural networks, function approximation, nonstationary environments, or insufficient exploration.

SARSA is on-policy:

Q(S_t,A_t) ← Q(S_t,A_t) + α[R_{t+1}+γQ(S_{t+1},A_{t+1})−Q(S_t,A_t)].

It learns the value of the policy actually followed, including its exploratory actions. Consequently, SARSA can prefer safer routes in some cliff-like tasks, whereas Q-learning may learn a riskier optimal route under its nominal greedy policy.

Exploration

A greedy policy always selects the action with the largest estimated value and may never discover better actions. Common strategies include:

• ε-greedy: choose a random action with probability ε and a greedy action otherwise. A decaying ε can provide early exploration and later exploitation.
• Boltzmann or softmax selection: assign action probabilities proportional to exp(Q(s,a)/τ), where τ controls exploration. Numerical stabilization, such as subtracting the maximum Q value before exponentiation, is important.
• Upper-confidence approaches: favor actions with high estimated value and high uncertainty.
• Entropy regularization: encourage stochastic policies during learning.

Exploration is not automatically harmless. In a medical, industrial, or safety-critical environment, random experimentation can injure people or damage equipment. Safe exploration uses constraints, simulators, conservative policies, human approval, or risk-sensitive objectives.

4. Deep Reinforcement Learning

Tabular methods fail when states are high-dimensional or continuous. Deep RL uses a neural network to approximate V(s), Q(s,a), or a policy. This introduces instability because experience samples are correlated, targets change during learning, and the same network may both predict targets and generate actions.

Deep Q-Networks

For discrete actions, a deep Q-network (DQN) approximates Q(s,a;θ). A typical loss for a nonterminal transition is

L(θ) = [r + γ max_{a'}Q(s',a';θ^−) − Q(s,a;θ)]²,

where θ are online-network parameters and θ^− are target-network parameters. The target network is held fixed for several updates and periodically copied from the online network. Experience replay stores transitions (s,a,r,s',done) and trains on randomly sampled minibatches. Replay reduces temporal correlation and improves data reuse.

A practical target is

y = r + γ(1-done) max_{a'}Q(s',a';θ^-).

The done mask prevents bootstrapping beyond a true terminal state. Care is required when an episode ends because of a time limit rather than the underlying task reaching a terminal condition; treating truncation as termination can bias targets.

DQN is suitable for moderate discrete action spaces. Double DQN uses the online network to select the maximizing action and the target network to evaluate it, reducing maximization bias:

y = r + γQ(s', argmax_{a'}Q(s',a';θ); θ^-).

Prioritized replay samples transitions with large estimated learning errors more often, but importance-sampling corrections may be needed to reduce sampling bias. Dueling architectures separately estimate state value and relative action advantages, which can help when many actions have similar effects.

Implementation Pattern

In either TensorFlow or PyTorch, a DQN implementation normally contains: a neural network, replay buffer, target network, optimizer, action-selection function, training step, checkpointing, and evaluation code. Pseudocode is:

1. Initialize online network Q_θ, target network Q_θ−, and replay buffer.

2. For each episode, reset the environment.

3. Select an ε-greedy action and observe next state, reward, and termination information.

4. Store the transition.

5. Once the buffer has enough samples, compute masked target values, mean-squared or Huber loss, backpropagate, and clip gradients if necessary.

6. Periodically copy θ to θ− and evaluate without exploration.

A PyTorch training fragment is conceptually:

```python

q = online(states).gather(1, actions)

with torch.no_grad():

next_q = target(next_states).max(1).values

y = rewards + gamma (1 - terminated.float()) next_q

loss = torch.nn.functional.smooth_l1_loss(q.squeeze(1), y)

optimizer.zero_grad()

loss.backward()

torch.nn.utils.clip_grad_norm_(online.parameters(), 10.0)

optimizer.step()

```

The code assumes tensors have compatible shapes and that `terminated` distinguishes genuine terminal states. A production implementation should also handle truncation, device placement, reproducible seeds, observation normalization, and evaluation separately from training.

5. Policy-Based and Actor–Critic Methods

Value methods learn what actions are good and derive a policy. Policy-based methods directly parameterize π_θ(a|s). They are natural for stochastic policies and continuous actions.

The objective is

J(θ)=E_{τ∼π_θ}[G_0],

where τ denotes a trajectory. The policy-gradient theorem gives, under standard differentiability and regularity assumptions,

∇_θJ(θ)=E[Σ_t ∇_θ log π_θ(A_t|S_t) G_t].

The likelihood-ratio form avoids differentiating through the environment dynamics. A simple REINFORCE algorithm samples complete episodes, computes returns, and updates θ in the direction of ∇ log π times the return. It is unbiased under appropriate sampling assumptions but often has high variance.

A baseline b(s) that does not depend on the selected action can reduce variance without changing the expected gradient:

∇_θJ(θ)=E[∇_θ log π_θ(A_t|S_t)(G_t−b(S_t))].

Using a learned value V_φ(s) gives the advantage estimate A_t≈G_t−V_φ(S_t). Actor–critic methods use an actor to represent π_θ and a critic to estimate V_φ or Q_φ. A one-step advantage can be estimated by

A_t ≈ R_{t+1}+γV_φ(S_{t+1})−V_φ(S_t).

Actor–critic methods generally trade some bias for lower variance. Generalized advantage estimation further combines multi-step estimates through a parameter controlling the bias–variance trade-off.

For continuous control, stochastic policies often output the mean and standard deviation of a Gaussian distribution, then sample an action and apply environment-specific bounds. Squashing a Gaussian through tanh changes the probability density; algorithms must include the corresponding log-probability correction when optimizing entropy or likelihood terms.

Modern methods such as proximal policy optimization constrain policy updates using a clipped probability ratio. The practical motivation is to avoid excessively large policy changes, although clipping is not a guarantee of safety or monotonic improvement. Soft actor–critic maximizes expected reward plus an entropy term and is widely used for continuous action spaces.

6. Simulation, Offline Learning, and Evaluation

Simulation is valuable when real interaction is expensive, slow, or dangerous. A simulator permits millions of trials, controlled randomization, and testing of rare events. However, a policy may exploit simulator errors, unrealistic dynamics, or reward loopholes. This is known as the sim-to-real gap. Domain randomization, system identification, uncertainty modeling, and real-world validation can reduce—but not eliminate—the gap.

Offline RL learns from a fixed dataset D of previously collected transitions. It is useful when online exploration is unacceptable, such as healthcare or industrial control. The central difficulty is distribution shift: the learned policy may choose actions poorly represented in D. Standard off-policy algorithms can extrapolate inaccurately beyond the data support.

Conservative offline methods penalize unsupported actions or constrain the learned policy toward behavior observed in the dataset. Dataset quality matters: coverage, logging accuracy, action-selection context, missingness, and outcome measurement all affect validity. Offline evaluation may use importance sampling or model-based estimators, but these can have high variance or model bias. A strong baseline, uncertainty intervals, and limited prospective testing are essential.

RL evaluation must separate training and evaluation episodes and use multiple random seeds. Report mean performance and variability, not only the best run. Relevant metrics include cumulative return, success rate, constraint violations, worst-case or lower-quantile outcomes, latency, energy use, fairness measures, and robustness under changed conditions. Statistical comparisons should use independent evaluation trajectories and confidence intervals or appropriate hypothesis tests.

7. Safety, Reward Design, and Governance

Reward hacking occurs when an agent maximizes the specified reward while violating the designer’s intent. A delivery robot might obtain high reward by repeatedly selecting easy deliveries while avoiding difficult but important ones. Proxy rewards can also produce discrimination, privacy loss, unsafe shortcuts, or excessive resource consumption.

Safety should be represented explicitly. A constrained MDP augments the reward objective with costs C_i and constraints such as

E_π[Σ_t γ^t C_i,t] ≤ d_i,

where d_i is an acceptable budget. Lagrangian methods optimize reward minus weighted constraint costs, but tuning multipliers does not guarantee that every trajectory is safe. Hard action shields, runtime monitors, fallback controllers, reachability analysis, and human override are stronger safeguards.

Important governance practices include:

• define the objective and unacceptable behaviors before training;
• maintain logs of observations, actions, rewards, model versions, and overrides;
• test distribution shifts, adversarial conditions, and rare failures;
• protect personally identifiable and sensitive data;
• conduct impact assessments for recommender and allocation systems;
• provide auditability and meaningful human oversight;
• establish rollback and incident-response procedures.

In recommendation, maximizing clicks can amplify sensational content or create filter bubbles. In resource allocation, an apparently efficient policy may systematically underserve disadvantaged groups. Fairness is context-dependent, and incompatible statistical criteria may prevent satisfying every desired notion simultaneously. Human and legal review is therefore necessary in addition to technical optimization.

8. Applications

Control and Robotics

RL can control locomotion, manipulation, energy systems, and traffic signals. The action may be a torque, velocity, or set point. Model-based controllers often provide stability and sample efficiency, while RL can optimize behaviors difficult to model. A practical hybrid uses a conventional safety controller around an RL policy.

Recommendation and Personalization

Recommendations are sequential because today’s content affects future preferences, retention, and exposure. Short-term click reward can conflict with long-term satisfaction. Contextual bandits may be sufficient when actions do not substantially change future states; full RL is justified when long-term user dynamics matter. Consent, privacy, diversity, and exposure fairness should be treated as design requirements.

Resource Allocation

Cloud scheduling, warehouse operations, inventory, and network routing can be formulated as MDPs. Actions allocate limited resources; rewards combine service quality, cost, delay, and energy. Constraints and nonstationarity are central: demand patterns change, and a policy that performs well under average load may fail during peaks.

9. Detailed Applied Case: Illustrative Data-Center Cooling Control

This is a fictional illustrative case, not a report of a particular company.

Background and Problem

A data center wants to reduce cooling electricity while maintaining safe server inlet temperatures. Measurements arrive every five minutes. The controller may adjust cooling set points and airflow, but abrupt changes can damage equipment or reduce service reliability.

MDP Formulation

The state contains recent temperature and humidity measurements, rack utilization, outside weather, cooling-unit status, and previous control settings. Including recent measurements helps approximate hidden thermal dynamics. Actions are discrete combinations of set-point and airflow adjustments. The reward is designed as negative energy consumption minus penalties for temperature violations and excessive actuator changes. A separate safety constraint limits the probability and duration of unsafe temperatures.

Analysis and Approach

Historical logs are first cleaned and checked for sensor failures, missing actions, and changes in equipment. A calibrated simulator is built from physical and data-driven models. Offline conservative training is performed using logged transitions, while unsafe actions are filtered by a safety controller. Domain randomization varies weather, load, and equipment efficiency. The final policy is evaluated against the existing controller on unseen simulated scenarios, including sensor faults and peak demand.

Deployment begins in shadow mode: the RL policy recommends actions but does not control equipment. Engineers compare recommendations with the baseline and inspect explanations, constraint margins, and counterfactual outcomes. A limited pilot then uses a human-approved action envelope and automatic fallback to the conventional controller.

Outcome

In this illustrative scenario, evaluation shows lower simulated energy use while meeting the temperature constraint in ordinary conditions. Stress tests reveal occasional instability after rapid load changes. The deployment is therefore postponed until the observation window is extended, the action rate is limited, and the fallback monitor is improved. This is a successful engineering outcome even though the policy is not immediately deployed: identifying unsafe behavior before full automation is a primary objective.

Learning Points

The case demonstrates that state design, reward construction, simulator validation, offline distribution shift, safety constraints, baseline comparison, and staged deployment matter as much as neural-network architecture. It also shows that operational RL is a control-system project, not merely a model-training exercise.

10. Best Practices and Contemporary Directions

Use the simplest formulation that captures the decision process. Start with a random, heuristic, and strong conventional baseline. Verify reward calculations independently, inspect trajectories rather than only aggregate returns, and test sensitivity to γ, reward scales, seeds, and observation changes. Normalize inputs where appropriate, clip or robustly scale rewards cautiously, and preserve enough information to reproduce every experiment.

Choose value-based learning for manageable discrete actions, policy-gradient or actor–critic methods for stochastic or continuous control, and model-based or hybrid methods where interaction is costly. Consider contextual bandits when actions have negligible long-term effects. Never infer that a larger neural network or a higher training return implies safer or more useful behavior.

Current research emphasizes offline and batch RL, model-based planning, uncertainty-aware exploration, multi-agent learning, hierarchical policies, safe and risk-sensitive optimization, and foundation models for agents. Open problems include reliable long-horizon credit assignment, causal representation of interventions, robust generalization, human preference alignment, and verifiable safety.

Chapter Summary

Reinforcement learning models repeated interaction between an agent and an environment. An MDP formalizes states, actions, transitions, rewards, and discounting under the Markov assumption. Return and value functions quantify long-term consequences, while Bellman equations support dynamic programming and temporal-difference learning. Q-learning is an off-policy value-based method; SARSA is on-policy. Deep Q-networks use replay and target networks to stabilize neural approximation. Policy gradients directly optimize stochastic policies, and actor–critic methods combine policy learning with value estimation.

Exploration is necessary but can be dangerous. Simulation increases sample availability but introduces model bias and sim-to-real risk. Offline RL avoids unsafe online experimentation but faces distribution shift and limited action coverage. Reliable evaluation requires independent trials, variability estimates, safety metrics, and strong baselines. Applications in control, recommendation, and resource allocation require careful reward design, constraints, privacy, fairness, monitoring, and staged deployment. The most effective RL practice combines algorithms with system engineering, domain knowledge, governance, and human oversight.

Key Terms

Markov decision process: A model of sequential decisions defined by states, actions, transitions, rewards, and discounting.

Markov property: The current state contains sufficient information to predict the next state and reward given an action.

Agent: The decision-making system that selects actions.

Environment: The external system with which the agent interacts.

Policy: A rule or probability distribution mapping states to actions.

Return: Discounted cumulative future reward.

Discount factor: A parameter determining the relative importance of future rewards.

Value function: Expected return from a state under a policy.

Action-value function: Expected return from a state-action pair.

Bellman equation: A recursive relationship expressing value through immediate reward and future value.

Temporal-difference error: The difference between a bootstrapped target and a current value estimate.

Q-learning: An off-policy algorithm that learns optimal action values.

SARSA: An on-policy temporal-difference control algorithm.

Exploration: Trying actions to gain information about their consequences.

Experience replay: Training from stored transitions sampled from a replay buffer.

Target network: A slowly updated network used to stabilize deep Q-learning targets.

Policy gradient: A method that optimizes policy parameters using gradients of expected return.

Actor–critic: An architecture with a policy actor and a value-function critic.

Offline reinforcement learning: Learning from a fixed dataset without new environment interaction.

Reward hacking: Optimizing a stated reward in a way that violates the intended objective.

Review Questions

Short Answer Questions

1. What is the Markov property, and why can an inadequate state representation violate it?

2. Distinguish the state-value function from the action-value function.

3. Why do DQN algorithms use experience replay and a target network?

4. What is the difference between on-policy and off-policy learning?

5. Why can reward design create safety and governance problems?

Long Answer Questions

1. Explain how an MDP is formulated and derive the Bellman expectation equation for a policy.

2. Compare Q-learning, SARSA, and deep Q-learning in terms of targets, exploration, assumptions, and limitations.

3. Explain the policy-gradient theorem and how actor–critic methods reduce variance.

4. Discuss the opportunities and risks of using simulation for reinforcement-learning control.

5. Analyze why offline RL is attractive in safety-critical settings and why distribution shift makes it difficult.

Analytical / Application Questions

1. Design an MDP for warehouse robot routing. Specify the state, action, transition, reward, discount factor, terminal conditions, and at least two safety constraints.

2. A recommender receives a high immediate click reward for sensational content but users later disengage. Explain how the reward and evaluation design should be changed to represent long-term value and user well-being.

3. Propose a staged deployment plan for an RL controller in an industrial plant. Include simulation, offline evaluation, safety monitoring, human oversight, fallback behavior, and criteria for progressing between stages.

Multiple Choice Questions with Answers

Question 1

Which expression defines the discounted return at time t?

A. The immediate reward only

B. The sum of future rewards weighted by powers of the discount factor

C. The maximum action probability in the current state

D. The average reward divided by the episode length

Correct Answer: B. The sum of future rewards weighted by powers of the discount factor

Question 2

What does the Markov property require?

A. All states must have equal probability

B. Rewards must always be positive

C. The current state must contain sufficient information for predicting the next state and reward given the action

D. The agent must use a deterministic policy

Correct Answer: C. The current state must contain sufficient information for predicting the next state and reward given the action

Question 3

Which algorithm uses a greedy next-action value in its update even when behavior explores?

A. Q-learning

B. REINFORCE only

C. SARSA

D. Monte Carlo policy evaluation only

Correct Answer: A. Q-learning

Question 4

What is the main purpose of a target network in DQN?

A. To increase the number of available actions

B. To provide a relatively stable network for computing learning targets

C. To remove the need for rewards

D. To guarantee safe exploration

Correct Answer: B. To provide a relatively stable network for computing learning targets

Question 5

In ε-greedy exploration, what happens with probability ε?

A. The policy always selects the highest-valued action

B. The episode terminates

C. An exploratory action is selected, commonly at random

D. The discount factor becomes zero

Correct Answer: C. An exploratory action is selected, commonly at random

Question 6

Which statement best describes policy-gradient learning?

A. It directly adjusts policy parameters using an estimate of the gradient of expected return

B. It requires a complete known transition model

C. It can only be used with tabular states

D. It never uses stochastic policies

Correct Answer: A. It directly adjusts policy parameters using an estimate of the gradient of expected return

Question 7

Why is offline reinforcement learning vulnerable to distribution shift?

A. It cannot store transitions

B. The learned policy may select actions poorly represented in the fixed dataset

C. It always uses a discount factor of one

D. It has no reward signal

Correct Answer: B. The learned policy may select actions poorly represented in the fixed dataset

Question 8

Which situation is an example of reward hacking?

A. An agent follows a safety constraint and receives the intended reward

B. An agent maximizes a proxy reward through behavior that violates the designer’s real objective

C. A critic estimates a value function

D. A simulator randomizes weather conditions

Correct Answer: B. An agent maximizes a proxy reward through behavior that violates the designer’s real objective

Question 9

For a genuine terminal transition in a DQN target, what should happen to the bootstrapped future-value term?

A. It should be multiplied by zero

B. It should be doubled

C. It should be replaced by the action probability

D. It should be divided by the reward

Correct Answer: A. It should be multiplied by zero

Question 10

Which practice is most appropriate before deploying an RL controller in a safety-critical system?

A. Deploy random exploration directly on the equipment

B. Evaluate only the single best training episode

C. Use staged testing, safety monitoring, and a verified fallback controller

D. Remove all constraints to maximize return

Correct Answer: C. Use staged testing, safety monitoring, and a verified fallback controller