Planning, Reinforcement Learning and Autonomous Decision-Making

Introduction and Context

An intelligent system is useful not merely because it can perceive or predict, but because it can select and execute actions that advance its objectives. Planning and reinforcement learning provide two major approaches to this problem. Planning traditionally assumes that an agent possesses an explicit model of the world, including states, actions, transition rules, and goals. Reinforcement learning (RL), by contrast, emphasizes learning effective behavior through interaction and feedback, often when the model is incomplete or unavailable.

Autonomous decision-making combines these perspectives. A self-driving vehicle, industrial robot, recommender system, or energy-management controller must infer relevant state, anticipate consequences, compare alternatives, balance immediate and long-term outcomes, and act under uncertainty. The decision problem is also shaped by reward design, safety constraints, computational resources, other agents, and ethical requirements.

This chapter develops classical and probabilistic planning, Markov decision processes, value and policy methods, reinforcement-learning algorithms, exploration, reward design, and multi-agent learning. It also evaluates their assumptions and practical limitations.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Distinguish classical planning from probabilistic planning and reinforcement learning.

2. Formulate a sequential decision problem as a Markov decision process.

3. Derive and interpret value functions, Bellman equations, and policies.

4. Compare dynamic programming, Monte Carlo, temporal-difference, and policy-gradient methods.

5. Design exploration strategies and identify common reward-design failures.

6. Explain the effects of partial observability, function approximation, and uncertainty.

7. Analyze multi-agent learning and its implications for autonomous systems.

8. Evaluate safety, governance, and deployment considerations in RL systems.

1. Planning as Deliberative Decision-Making

A planning problem specifies an initial state, a set of actions, a model of how actions change the world, and a desired objective. In classical planning, the world is commonly represented using symbolic predicates. For example, a warehouse robot may have predicates such as `At(robot, loading_dock)`, `Holding(robot, package)`, and `Available(dock)`. An action has:

• Preconditions: facts that must hold before execution.
• Effects: facts made true or false after execution.
• Cost: a numerical measure such as time, energy, or risk.

A plan is a sequence of actions whose preconditions are satisfied successively and whose final state satisfies the goal. Search algorithms such as breadth-first search, uniform-cost search, greedy best-first search, and A can generate plans. A evaluates a node using

f(n) = g(n) + h(n),

where g(n) is the cost already incurred and h(n) is an estimate of the remaining cost. If h is admissible—never greater than the true remaining cost—and costs are non-negative, A* is optimal under standard conditions. The quality of a plan therefore depends not only on search but also on the representation and heuristic.

Classical planning is attractive because its plans are often interpretable and its assumptions are explicit. It is effective in structured environments such as logistics, configuration, scheduling, and automated workflow management. However, exact symbolic models may be unrealistic in environments with noisy sensors, uncertain action outcomes, continuous dynamics, or unanticipated events.

1.1 Planning formalisms

State-space planning searches over complete world states. In forward planning, actions are applied from the initial state. In backward planning, the system works backward from the goal and identifies actions that could achieve it. Partial-order planning represents only necessary ordering constraints, allowing independent actions to remain unordered.

Hierarchical task network planning decomposes an abstract task into increasingly concrete subtasks. This can reduce search and encode domain expertise, although it may fail when the hierarchy excludes a valid solution. Planning graphs and satisfiability-based planning transform reachability into structured computational problems. These methods illustrate an important engineering principle: planning performance is strongly influenced by representation.

1.2 Probabilistic planning

Probabilistic planning represents uncertainty explicitly. An action may lead to several possible outcomes, each with a probability. A stochastic transition model is written as

P(s' | s, a),

meaning the probability that action a causes a transition from state s to state s'. A plan is no longer simply a sequence; it may be a contingent policy that specifies what to do after different observations or outcomes.

Probabilistic planning includes Markov decision processes when the state is fully observed and partially observable Markov decision processes (POMDPs) when it is not. It may also use belief-state planning, in which the agent plans over probability distributions describing its uncertainty about the actual state. This is computationally demanding, but it is conceptually appropriate for diagnosis, robotics, medical decisions, and operations under uncertainty.

2. Markov Decision Processes

A finite Markov decision process is commonly defined by the tuple (S, A, P, R, γ), where:

• S is the set of states.
• A is the set of actions, or A(s) is the set available in state s.
• P(s'|s,a) is the transition probability.
• R(s,a,s') is the immediate reward.
• γ is the discount factor, usually 0 ≤ γ < 1.

The Markov property states that, conditional on the current state and action, the distribution of the next state does not depend on the full history:

P(S_{t+1}|S_t,A_t,S_{t-1},A_{t-1},...) = P(S_{t+1}|S_t,A_t).

This is a modeling assumption. A state representation that omits relevant history is not Markov, even if the underlying physical process is Markov. For example, the current location of a robot may not be sufficient if battery temperature, previous actions, or a carrying status affects future outcomes.

A policy π specifies action selection. A stochastic policy is π(a|s), the probability of choosing a in state s; a deterministic policy is π(s). The discounted return from time t is

G_t = R_{t+1} + γR_{t+2} + γ²R_{t+3} + ... .

The discount factor expresses both temporal preference and a technical device for ensuring finite returns in continuing tasks. A lower γ emphasizes immediate rewards; a value close to one gives greater importance to distant consequences. Discounting should not be selected mechanically: in safety-critical or long-horizon applications it can create unacceptable short-term bias.

2.1 Value functions and Bellman equations

The state-value function under policy π is

V^π(s) = E_π[G_t | S_t=s].

The action-value function is

Q^π(s,a) = E_π[G_t | S_t=s,A_t=a].

Using the Markov property, the Bellman expectation equations are

V^π(s) = Σ_a π(a|s) Σ_{s'} P(s'|s,a)[R(s,a,s') + γV^π(s')],

and

Q^π(s,a) = Σ_{s'} P(s'|s,a)[R(s,a,s') + γΣ_{a'}π(a'|s')Q^π(s',a')].

These equations express a recursive decomposition: the value of a decision equals its expected immediate reward plus the discounted value of its successor. The optimal functions are

V(s) = max_a Σ_{s'}P(s'|s,a)[R(s,a,s') + γV(s')],

and

Q(s,a) = Σ_{s'}P(s'|s,a)[R(s,a,s') + γ max_{a'}Q(s',a')].

An optimal deterministic policy can be selected by choosing an action that maximizes Q*(s,a), provided the usual finite discounted MDP assumptions hold.

Worked Example: Expected action value

Suppose an agent in state s chooses action a. With probability 0.7 it receives reward 5 and reaches state s1, whose value is 10. With probability 0.3 it receives reward 1 and reaches state s2, whose value is 4. Let γ = 0.9. Then

Q(s,a) = 0.7[5 + 0.9(10)] + 0.3[1 + 0.9(4)]

= 0.7(14) + 0.3(4.6)

= 9.8 + 1.38

= 11.18.

The value is an expectation over possible outcomes, not the reward from the most likely outcome alone. This distinction is central to risk-sensitive decision-making.

3. Solving Known MDPs

When the transition and reward models are known, dynamic programming can compute or improve policies.

3.1 Policy evaluation and iteration

Policy evaluation repeatedly applies the Bellman expectation equation to estimate V^π. For a finite discounted MDP, iterative evaluation converges under standard conditions. Policy iteration alternates between evaluation and improvement:

1. Start with an arbitrary policy.

2. Evaluate its value function.

3. Improve the policy by selecting actions with greater one-step look-ahead value.

4. Repeat until the policy is stable.

Policy iteration often converges in relatively few improvement rounds, but each evaluation may be costly.

Value iteration combines truncated evaluation and improvement:

V_{k+1}(s) = max_a Σ_{s'}P(s'|s,a)[R(s,a,s') + γV_k(s')].

For finite states, bounded rewards, and γ < 1, the Bellman optimality operator is a contraction under the maximum norm, which supports convergence to V*. In practice, stopping criteria may be based on the maximum change between successive value estimates or on policy stability.

3.2 Planning versus learning

Planning uses a model to reason about consequences; learning estimates behavior or the model from experience. The boundary is not absolute. Model-based RL learns a transition and reward model and then plans with it. Model-free RL learns values or policies without explicitly constructing a model. Hybrid approaches can exploit known physics, simulators, demonstrations, and online data.

4. Reinforcement Learning from Experience

In reinforcement learning, an agent interacts with an environment. At time t it observes state S_t, selects action A_t, receives reward R_{t+1}, and observes S_{t+1}. The objective is to maximize expected cumulative return while learning from data.

4.1 Monte Carlo and temporal-difference learning

Monte Carlo methods update estimates after an episode terminates, using the observed return. For a state value, a generic sample update is

V(S_t) ← V(S_t) + α[G_t − V(S_t)],

where α is a learning rate and G_t is the sampled return. Monte Carlo methods make few assumptions about transitions but require complete episodes and can have high variance.

Temporal-difference (TD) learning updates before the episode ends by bootstrapping from an estimate of the next state:

V(S_t) ← V(S_t) + α[R_{t+1} + γV(S_{t+1}) − V(S_t)].

The bracketed term is the TD error. TD methods generally learn online and can be more data-efficient, but their estimates depend on the quality of current predictions.

4.2 SARSA and Q-learning

SARSA is an on-policy control method. Its update is

Q(S_t,A_t) ← Q(S_t,A_t) + α[R_{t+1} + γQ(S_{t+1},A_{t+1}) − Q(S_t,A_t)].

It learns the value of the behavior policy, including its exploration choices.

Q-learning is an off-policy method:

Q(S_t,A_t) ← Q(S_t,A_t) + α[R_{t+1} + γ max_a Q(S_{t+1},a) − Q(S_t,A_t)].

The target uses the greedy next action even if the behavior policy explores. Under tabular conditions, suitable learning rates, sufficient exploration, and standard assumptions, Q-learning converges to Q* with probability one. These conditions are not generally preserved with nonlinear function approximation, partial observability, or unstable data distributions.

Worked Example: One Q-learning update

Assume Q(s,a)=4, the observed reward is 3, γ=0.9, and the largest estimated value in the next state is 6. With α=0.2, the target is

3 + 0.9(6) = 8.4.

The update is

Q_new = 4 + 0.2(8.4 − 4) = 4 + 0.88 = 4.88.

The estimate moves toward, but does not immediately equal, the target.

4.3 Deep reinforcement learning

When state spaces are large, a table is impractical. Function approximators, particularly neural networks, can estimate V(s), Q(s,a), or a policy. Deep Q-networks use mechanisms such as experience replay and a target network to reduce correlations and moving-target instability. Actor–critic methods maintain a policy (actor) and a value estimator (critic). Policy-gradient methods optimize an expected objective by adjusting policy parameters; they are useful for continuous actions but may have high variance.

Proximal policy optimization, soft actor–critic, and deterministic actor–critic methods are widely used families, but no algorithm is universally superior. Performance depends on reward structure, action space, simulator fidelity, data availability, and safety requirements. Deep RL should therefore be treated as an engineering system, not merely as a choice of optimizer.

5. Exploration, Exploitation, and Uncertainty

An agent must exploit actions currently believed to be good while exploring actions whose consequences are uncertain. Excessive exploitation can trap learning in a poor policy; excessive exploration can waste resources or create danger.

An ε-greedy policy selects a greedy action with probability 1−ε and a random action with probability ε. A decaying ε schedule may begin with broad exploration and become more exploitative. Softmax action selection assigns probabilities according to estimated action values and a temperature parameter. Upper-confidence methods add an uncertainty bonus, favoring actions with high estimated value or limited experience.

Exploration is not only random action selection. It may involve curiosity bonuses, information gain, intrinsic motivation, Bayesian model uncertainty, optimistic initialization, demonstrations, or strategically designed experiments. In physical systems, safe exploration may restrict actions to a certified region or use a backup controller.

The exploration problem is especially difficult in sparse-reward environments. If success occurs only after a long sequence, random exploration may almost never discover it. Curriculum learning, demonstrations, subgoals, reward shaping, and model-based search can improve sample efficiency.

6. Reward Design and Decision Objectives

Reward is an operational specification of what the system optimizes. A poorly designed reward can produce behavior that is technically optimal but socially or operationally unacceptable. This is often called reward hacking or specification gaming.

A reward should be:

• aligned with the actual objective rather than a convenient proxy;
• measurable with acceptable reliability;
• resistant to manipulation by the agent;
• balanced across safety, performance, fairness, and resource use;
• auditable by domain experts.

Potential-based shaping can preserve optimal policies under standard discounted MDP assumptions when the shaping reward has the form

F(s,a,s') = γΦ(s') − Φ(s),

where Φ is a potential function. This result does not mean every intuitive shaping signal is safe. Incorrect shaping can change the optimal policy, create shortcuts, or encourage behavior that exploits measurement artifacts.

Multi-objective decision-making may use a weighted reward such as

R = w_1R_1 + w_2R_2 + ... + w_nR_n,

but weights encode value judgments and may hide unacceptable trade-offs. In safety-critical applications, constraints are often preferable to simple penalties. A constrained formulation may maximize expected return subject to limits on collision probability, emissions, financial loss, or demographic disparity.

7. Partial Observability and Belief-State Decision-Making

Many environments do not expose the true state. A robot may receive noisy sensor readings; a financial system may not observe market regimes; a medical agent may not know a patient’s latent condition. A POMDP adds observations O and an observation model P(o|s',a).

The agent can maintain a belief state b(s), a probability distribution over possible states. After action a and observation o, Bayesian belief updating is conceptually represented by

b'(s') ∝ P(o|s',a) Σ_s P(s'|s,a)b(s),

where proportionality means that the result is normalized so probabilities sum to one. The belief state is a sufficient statistic for history under the POMDP model, but the resulting state space is continuous and often computationally difficult.

Approximate methods include point-based value iteration, particle filters, recurrent neural networks, predictive state representations, and planning over limited horizons. A recurrent policy can summarize history, but it does not automatically guarantee correct belief inference.

8. Multi-Agent Learning

In multi-agent settings, the environment changes as other agents learn. A single-agent MDP assumption may fail because the transition distribution is no longer stationary from one learner’s perspective. Agents may cooperate, compete, or combine both objectives.

Important settings include:

• cooperative teams with shared rewards;
• competitive games with opposing rewards;
• mixed-motive environments with partially aligned interests;
• mechanisms in which agents communicate or negotiate.

Centralized training with decentralized execution allows agents to use global information during learning but act using local observations at deployment. Value decomposition methods factor a team value into individual utilities, while opponent modeling attempts to predict other agents’ policies. Multi-agent credit assignment is difficult because a team reward does not reveal which agent caused success or failure.

Self-play can generate strong behavior in competitive domains, but it may overfit to the training population. Communication may improve coordination but introduces bandwidth, security, and protocol-learning issues. Strategic manipulation, collusion, free-riding, and emergent conventions must be evaluated explicitly.

9. A Detailed Applied Case: Autonomous Warehouse Fleet

The following is an illustrative case, not a report of a specific commercial deployment.

Background

A distribution center uses mobile robots to transport shelves and packages. Orders arrive continuously, aisles may become blocked, batteries require charging, and human workers share the environment. The objective is to increase throughput without compromising worker safety or equipment reliability.

Problem

A purely classical planner can generate efficient routes when the map is accurate, but it may perform poorly when congestion, delays, or temporary obstacles occur. A purely model-free RL controller may discover useful dispatching behavior but could explore unsafe routes and may require an unrealistic volume of operational data.

Analysis

The system is decomposed into levels. A symbolic planner handles task allocation and hard constraints such as restricted zones. A probabilistic route planner estimates travel time and blockage risk. An RL policy selects among safe, already validated route alternatives and charging actions. The state includes robot location, battery level, assigned task, local congestion, and estimated human proximity. The action set contains route selection, waiting, charging, and task reassignment.

The reward combines delivery progress, delay penalties, energy consumption, and safety violations. Safety violations are not treated merely as a large negative reward; a runtime shield prevents actions that violate separation or speed constraints. Historical logs and simulation provide initial training data, followed by carefully monitored online adaptation.

Approach and solution

The design uses constrained action selection. A learned value function ranks feasible actions, while a model-based predictor estimates congestion. Exploration is restricted to actions that satisfy collision-avoidance rules and battery reserves. A fallback controller returns robots to a safe state if sensors or communication fail. Performance is evaluated against rule-based and optimization-based baselines using throughput, mean delay, energy per delivery, near-miss rate, intervention rate, and worst-case behavior.

Outcome

In a credible evaluation, success would not be defined by average reward alone. The deployment decision would require statistically reliable improvement in throughput, no unacceptable increase in safety incidents, robustness to demand surges, and understandable operational behavior. If the learned policy improves average delivery time but increases rare high-severity failures, it should not be deployed without redesign.

Learning points

1. Hybrid architectures often provide better assurance than an unconstrained end-to-end learner.

2. Reward design must represent safety and operational constraints directly.

3. Simulation is valuable but can produce misleading confidence when the simulator omits human unpredictability.

4. Evaluation requires distribution-shift tests, not only random held-out episodes.

5. Human override, logging, rollback, and incident review are parts of the learning system.

10. Evaluation, Deployment, and Governance

RL performance should be evaluated across several dimensions:

• expected return and task success;
• sample efficiency and training cost;
• robustness to disturbances and distribution shift;
• safety violations and constraint satisfaction;
• fairness across affected groups;
• interpretability and operator trust;
• latency, energy use, and reliability.

Average return can conceal catastrophic tail outcomes. Quantile measures, worst-case analysis, risk-sensitive objectives, and stress testing are therefore important. Offline RL introduces a further challenge: the learner uses a fixed dataset and may select actions poorly represented in that data. Conservative methods and uncertainty estimates can reduce, but not eliminate, extrapolation error.

Governance should establish who defines objectives, who authorizes deployment, how models are monitored, and how incidents are reported. Systems handling personal, financial, employment, medical, or safety-related decisions may be subject to privacy, nondiscrimination, product-safety, and sector-specific regulation. Logging must support auditability without creating unnecessary privacy exposure. Human oversight should be meaningful rather than a nominal approval step.

11. Challenges and Contemporary Directions

Several unresolved issues shape current research and practice.

Generalization: Policies trained in one environment may fail under new layouts, users, weather, hardware, or demand patterns.

Sim-to-real transfer: Simulators are cheaper and safer than physical experiments, but model mismatch can cause unexpected behavior. Domain randomization, system identification, and real-world fine-tuning are common responses.

Long-horizon credit assignment: Delayed consequences make it difficult to identify which actions caused success or failure.

Offline and batch RL: Historical data may contain selection bias and few examples of good decisions. Learning from logs requires careful support and uncertainty analysis.

Safe and robust RL: Formal constraints, shielding, verified controllers, risk-sensitive objectives, and adversarial testing are increasingly integrated with learning.

Human-AI interaction: An autonomous agent must account for human intentions, trust calibration, intervention behavior, and the possibility that users adapt strategically to it.

Foundation models and agents: Language and multimodal models can support planning, tool use, and natural-language instruction, but generated plans require grounding, verification, and protection against prompt or tool manipulation. A language model’s fluent explanation is not evidence that its policy is safe or optimal.

Best Practices

1. Define the decision objective and unacceptable outcomes before selecting an algorithm.

2. Verify whether the proposed state representation is sufficiently Markov for the task.

3. Establish strong non-learning baselines.

4. Separate training, validation, stress testing, and deployment monitoring.

5. Use simulators and offline data, but quantify their mismatch with deployment conditions.

6. Treat exploration as a safety and experimental-design problem.

7. Prefer constraints, shields, and fallback policies for high-consequence actions.

8. Inspect reward channels for proxy failure and unintended incentives.

9. Report variance, tail risk, ablations, and failures—not only best-run average reward.

10. Maintain versioned data, models, policies, logs, and rollback procedures.

Chapter Summary

Classical planning searches for action sequences using explicit symbolic models, while probabilistic planning represents uncertain outcomes and may produce contingent policies. Markov decision processes formalize sequential decisions through states, actions, transition probabilities, rewards, and discounting. Value functions and Bellman equations provide the foundation for policy evaluation and optimal control.

Dynamic programming solves known MDPs; Monte Carlo and temporal-difference methods learn from experience; SARSA and Q-learning illustrate on-policy and off-policy control. Deep RL extends these ideas to large or continuous spaces but introduces instability, data demands, and safety concerns. Exploration must balance information gathering with exploitation and risk. Reward design is a central specification problem, and reward shaping can produce unintended behavior.

Partial observability requires belief-state or memory-based decision-making. Multi-agent learning adds non-stationarity, coordination, competition, credit assignment, and communication challenges. Practical autonomous systems usually benefit from hybrid architectures that combine planning, learning, constraints, monitoring, and human oversight. Evaluation must include robustness, safety, fairness, and tail outcomes rather than average reward alone.

Key Terms

Markov decision process: A formal model of sequential decision-making with states, actions, transitions, rewards, and discounting.

Markov property: The assumption that the current state contains all information needed to predict future states given an action.

Policy: A rule or probability distribution specifying actions in states.

Return: The cumulative, usually discounted, reward following a time step.

Value function: Expected return from a state under a specified policy.

Action-value function: Expected return from taking an action in a state and then following a policy.

Bellman equation: A recursive relationship expressing value through immediate reward and successor value.

Dynamic programming: Planning or control using a known model and recursive value computations.

Monte Carlo learning: Learning from complete sampled returns.

Temporal-difference learning: Learning by bootstrapping from an estimate of a successor state.

On-policy learning: Learning the value of the policy used to generate behavior.

Off-policy learning: Learning a target policy different from the behavior policy.

Exploration: Selecting uncertain or unfamiliar actions to obtain information.

Reward shaping: Adding learning signals intended to guide behavior toward an objective.

Reward hacking: Achieving high specified reward through unintended or undesirable behavior.

POMDP: A partially observable Markov decision process with hidden states and observations.

Belief state: A probability distribution representing uncertainty about the actual state.

Multi-agent reinforcement learning: Learning and decision-making involving multiple adaptive agents.

Safe exploration: Exploration restricted by safety rules, constraints, or backup mechanisms.

Sim-to-real transfer: Applying behavior learned in simulation to a physical or operational environment.

Review Questions

Short Answer Questions

1. What are the principal components of a Markov decision process?

2. Distinguish policy evaluation from policy improvement.

3. What is the difference between on-policy and off-policy learning?

4. Why can reward shaping create unintended behavior?

5. What is a belief state in a POMDP?

Long Answer Questions

1. Explain classical, probabilistic, and hierarchical planning, including their assumptions and limitations.

2. Derive the Bellman expectation and optimality equations and explain their significance.

3. Compare Monte Carlo learning, TD learning, SARSA, and Q-learning.

4. Discuss exploration strategies and their suitability for safety-critical applications.

5. Explain the principal challenges of multi-agent reinforcement learning.

Analytical / Application Questions

1. Design an MDP for hospital bed allocation. Define the state, actions, transition uncertainty, rewards, constraints, and evaluation metrics. Discuss which information may violate the Markov assumption.

2. A delivery robot receives a high reward for completing orders quickly but no penalty for entering restricted pedestrian zones. Analyze the likely failure mode and propose a revised objective and safety architecture.

3. Compare a model-based planner, tabular Q-learning, and a deep actor–critic system for warehouse control. Recommend a hybrid design and justify it using data availability, safety, observability, and deployment constraints.

Multiple Choice Questions with Answers

Question 1

Which condition is central to the Markov property?

A. The next state depends only on the complete history

B. The current state contains sufficient information about the past for predicting the future given the action

C. All actions have identical rewards

D. The environment must be deterministic

Correct Answer: B. The current state contains sufficient information about the past for predicting the future given the action

Question 2

In a discounted MDP, what does gamma represent?

A. The learning rate

B. The probability of exploration

C. The discount factor applied to future rewards

D. The transition probability

Correct Answer: C. The discount factor applied to future rewards

Question 3

Which algorithm uses the greedy next-action value in its update target while behavior may explore?

A. Q-learning

B. SARSA

C. Monte Carlo prediction only

D. Breadth-first search

Correct Answer: A. Q-learning

Question 4

Using the chapter’s worked Q-learning example, with Q=4, reward=3, gamma=0.9, next maximum Q=6, and alpha=0.2, what is the updated Q value?

A. 4.20

B. 4.80

C. 4.88

D. 8.40

Correct Answer: C. 4.88

Question 5

What is a principal weakness of Monte Carlo reinforcement learning?

A. It cannot use observed rewards

B. It generally requires an episode to reach a terminal point before updating from the return

C. It always assumes a known transition model

D. It cannot learn policies

Correct Answer: B. It generally requires an episode to reach a terminal point before updating from the return

Question 6

Which approach directly represents uncertainty about the hidden state in a POMDP?

A. A belief state

B. A deterministic action list only

C. A fixed discount factor only

D. A planning graph without observations

Correct Answer: A. A belief state

Question 7

What is reward hacking?

A. Reducing the learning rate during training

B. Achieving the specified reward through behavior that violates the intended objective

C. Using a stochastic policy

D. Estimating a transition probability

Correct Answer: B. Achieving the specified reward through behavior that violates the intended objective

Question 8

What is a key reason multi-agent environments can be non-stationary from one learner’s perspective?

A. The state space is always finite

B. Other agents change their policies while learning

C. Rewards are necessarily absent

D. All agents share identical observations

Correct Answer: B. Other agents change their policies while learning

Question 9

Under standard discounted MDP assumptions, which shaping form can preserve optimal policies?

A. F(s,a,s') = Phi(s) + Phi(s')

B. F(s,a,s') = gamma Phi(s') - Phi(s)

C. F(s,a,s') = gamma + Phi(s)

D. F(s,a,s') = R(s,a,s') squared

Correct Answer: B. F(s,a,s') = gamma Phi(s') - Phi(s)

Question 10

Why is average return insufficient for evaluating a safety-critical RL system?

A. Average return cannot be computed

B. It may conceal rare but catastrophic failures

C. It measures only transition probabilities

D. It guarantees fairness

Correct Answer: B. It may conceal rare but catastrophic failures