Reinforcement Learning and Sequential Decision-Making

Introduction

Reinforcement learning (RL) studies how an agent learns to make decisions through interaction with an environment. Unlike supervised learning, the agent is not given a correct action for every situation. Instead, it receives rewards or costs and must discover a policy that maximizes long-term performance. This makes RL particularly relevant to resource allocation, industrial control, networking, recommendation, robotics, energy management, and adaptive operations.

The central difficulty is temporal credit assignment: an action may have consequences many steps later. A second difficulty is the exploration–exploitation trade-off. The agent must exploit actions that currently appear effective while exploring alternatives that may be better. In real systems, decisions also face safety constraints, delayed observations, partial observability, changing dynamics, and limited or purely historical data.

This chapter develops the mathematical foundations of Markov decision processes, dynamic programming, temporal-difference learning, policy-gradient methods, actor–critic algorithms, and offline RL. It emphasizes assumptions, implementation choices, numerical reasoning, and limitations relevant to postgraduate IT engineering.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Formulate sequential decision problems as Markov decision processes (MDPs).

2. Derive and apply Bellman expectation and optimality equations.

3. Explain policy evaluation, policy improvement, value iteration, and temporal-difference learning.

4. Distinguish value-based, policy-based, and actor–critic methods.

5. Implement the main logic of policy gradients and actor–critic algorithms.

6. Analyze offline RL, distribution shift, extrapolation error, and conservative learning.

7. Design an RL solution for resource allocation or control while addressing safety, evaluation, and governance.

1. Sequential Decision Problems and MDPs

An MDP is defined by the tuple

\[

\mathcal{M}=(\mathcal{S},\mathcal{A},P,R,\gamma),

\]

where \(\mathcal{S}\) is the state space, \(\mathcal{A}\) the action space, \(P(s'|s,a)\) the transition probability, \(R(s,a,s')\) the reward, and \(\gamma\in[0,1)\) the discount factor. At time \(t\), the agent observes state \(S_t\), selects action \(A_t\), receives reward \(R_{t+1}\), and moves to \(S_{t+1}\).

The Markov property requires

\[

P(S_{t+1},R_{t+1}|S_0,A_0,\ldots,S_t,A_t)=P(S_{t+1},R_{t+1}|S_t,A_t).

\]

Thus, the current state contains all information needed to predict the future, although it may be necessary to construct the state from a history of observations. If the system is only partially observed, it is more appropriately modeled as a partially observable MDP (POMDP), often using a belief state or recurrent neural network.

A policy \(\pi(a|s)\) specifies the probability of selecting action \(a\) in state \(s\). A deterministic policy is written \(a=\pi(s)\). The discounted return from time \(t\) is

\[

G_t=\sum_{k=0}^{\infty}\gamma^kR_{t+k+1}.

\]

For episodic tasks, the sum ends at the terminal time. Discounting can represent time preference, uncertainty about the future, or a technical device ensuring finite returns. It must not be selected independently of operational requirements: a low \(\gamma\) favors immediate performance, whereas a high \(\gamma\) emphasizes long-term effects.

The state-value and action-value functions under policy \(\pi\) are

\[

V^\pi(s)=\mathbb{E}_\pi[G_t\mid S_t=s],

\]

\[

Q^\pi(s,a)=\mathbb{E}_\pi[G_t\mid S_t=s,A_t=a].

\]

The optimal functions are \(V^(s)=\max_\pi V^\pi(s)\) and \(Q^(s,a)=\max_\pi Q^\pi(s,a)\). A policy is optimal if it achieves these values.

Rewards, costs, and constraints

Many engineering systems are naturally specified using costs, such as energy consumption, latency, or packet loss. One can define reward as the negative cost, but scaling matters. If the reward is \(r_t=-c_t\), maximizing return minimizes discounted cost. In constrained RL, the objective may be

\[

\max_\pi J(\pi) \quad\text{subject to}\quad \mathbb{E}_\pi\left[\sum_t\gamma^t C_t\right]\leq d,

\]

where \(C_t\) is a safety or resource cost and \(d\) is a permitted budget. A Lagrangian approach optimizes \(J(\pi)-\lambda(C(\pi)-d)\), but constraint satisfaction is not automatically guaranteed during learning.

2. Bellman Equations and Dynamic Programming

The Bellman expectation equation follows by separating the first reward from the remainder of the return:

\[

V^\pi(s)=\sum_a\pi(a|s)\sum_{s',r}p(s',r|s,a)[r+\gamma V^\pi(s')].

\]

Similarly,

\[

Q^\pi(s,a)=\sum_{s',r}p(s',r|s,a)\left[r+\gamma\sum_{a'}\pi(a'|s')Q^\pi(s',a')\right].

\]

The Bellman optimality equations are

\[

V^(s)=\max_a\sum_{s',r}p(s',r|s,a)[r+\gamma V^(s')],

\]

\[

Q^(s,a)=\sum_{s',r}p(s',r|s,a)[r+\gamma\max_{a'}Q^(s',a')].

\]

These equations are fixed-point equations. For finite MDPs and \(\gamma<1\), the Bellman optimality operator is a contraction in the sup norm, so repeated application converges to the unique optimal value function.

Policy evaluation and policy iteration

Given a fixed policy, iterative policy evaluation updates

\[

V_{k+1}(s)=\sum_a\pi(a|s)\sum_{s',r}p(s',r|s,a)[r+\gamma V_k(s')].

\]

Policy improvement selects a greedy action:

\[

\pi_{new}(s)\in\arg\max_a\sum_{s',r}p(s',r|s,a)[r+\gamma V^\pi(s')].

\]

Policy iteration alternates evaluation and improvement. For finite discounted MDPs, each strict improvement increases the value of at least one state and the procedure terminates in an optimal policy, assuming exact evaluation and improvement.

Value iteration combines evaluation and improvement:

\[

V_{k+1}(s)=\max_a\sum_{s',r}p(s',r|s,a)[r+\gamma V_k(s')].

\]

It is often computationally cheaper but requires a known transition model and reward model.

Worked numerical example

Consider a finite discounted problem with \(\gamma=0.9\). Suppose a one-step calculation has already established the relevant continuation values. For state \(L\), action \(C\) yields immediate reward 2 and an expected next-state value of 10.7111. Therefore,

\[

Q(L,C)=2+0.9(10.7111)=11.6400\approx11.64.

\]

For state \(H\), action \(C\) yields immediate reward 3 and expected next-state value 10.5111:

\[

Q(H,C)=3+0.9(10.5111)=12.4600\approx12.46.

\]

For state \(H\), action \(A\) yields immediate reward 4 and expected next-state value 9.5111:

\[

Q(H,A)=4+0.9(9.5111)=12.5600\approx12.56.

\]

Thus, at state \(H\), action \(A\) is preferred to action \(C\), because \(12.56>12.46\). At state \(L\), the action must be compared with the other available action using the same Bellman calculation. The example illustrates that a larger immediate reward does not alone determine the best action; the transition-dependent continuation value is also essential.

3. Model-Free Value Learning

Dynamic programming requires a known model. In model-free RL, the agent learns from sampled transitions.

Monte Carlo learning

Monte Carlo evaluation uses complete returns. For a visit to state \(S_t\), the update is

\[

V(S_t)\leftarrow V(S_t)+\alpha[G_t-V(S_t)],

\]

where \(\alpha\in(0,1]\) is a learning rate. It is unbiased with respect to the sampled return under standard episodic assumptions, but it has high variance and must generally wait until an episode ends.

Temporal-difference learning

TD(0) bootstraps from the next state:

\[

\delta_t=R_{t+1}+\gamma V(S_{t+1})-V(S_t),

\]

\[

V(S_t)\leftarrow V(S_t)+\alpha\delta_t.

\]

For a terminal next state, the continuation term is zero. TD learning updates before episode termination and usually has lower variance than Monte Carlo learning, although its target is partly based on an estimate and is therefore biased during learning.

SARSA is an on-policy action-value method:

\[

\delta_t=R_{t+1}+\gamma Q(S_{t+1},A_{t+1})-Q(S_t,A_t),

\]

\[

Q(S_t,A_t)\leftarrow Q(S_t,A_t)+\alpha\delta_t.

\]

The next action is selected by the behavior policy, often \(\epsilon\)-greedy. Q-learning is off-policy:

\[

\delta_t=R_{t+1}+\gamma\max_{a'}Q(S_{t+1},a')-Q(S_t,A_t).

\]

It learns toward the greedy target even when behavior is exploratory. The max operator can produce overestimation because it preferentially selects noisy high estimates. Double Q-learning reduces this effect by separating action selection and evaluation across two estimates.

Eligibility traces and \(n\)-step methods

One-step TD and Monte Carlo learning are endpoints of a bias–variance continuum. An \(n\)-step target is

\[

G_{t:t+n}=R_{t+1}+\gamma R_{t+2}+\cdots+\gamma^{n-1}R_{t+n}+\gamma^nV(S_{t+n}).

\]

Eligibility traces combine multiple temporal horizons. TD(\lambda) uses a trace parameter \(\lambda\in[0,1]\); small values emphasize short bootstrapped targets and large values approach Monte Carlo behavior. In continuing tasks, trace decay and discounting must be selected carefully to prevent unstable accumulation.

4. Policy Gradients and Actor–Critic Learning

Value methods are convenient for small discrete action spaces, but continuous control or very large action spaces often favor direct policy optimization. Let the policy be differentiable and parameterized by \(\theta\), and define

\[

J(\theta)=\mathbb{E}_{\pi_\theta}[G_0].

\]

The policy-gradient theorem gives, under standard regularity conditions and an appropriate episodic or discounted occupancy distribution,

\[

\nabla_\theta J(\theta)=\mathbb{E}_{\pi_\theta}\left[\sum_{t=0}^{T-1}\gamma^t G_t\nabla_\theta\log\pi_\theta(A_t|S_t)\right]

\]

for the objective defined explicitly as \(J=\mathbb{E}[G_0]\), when trajectories are sampled from the initial-state distribution and the return formulation is used. The \(\gamma^t\) factor is required under this convention. Alternative policy-gradient statements absorb discounting into a discounted state-visitation distribution; those formulations must not be mixed without changing the objective or sampling interpretation.

A baseline \(b(S_t)\) independent of the selected action preserves unbiasedness:

\[

\mathbb{E}[\nabla_\theta\log\pi_\theta(A_t|S_t)b(S_t)\mid S_t]=0.

\]

Consequently, the advantage form is used:

\[

\nabla_\theta J(\theta)\approx\sum_t\gamma^t\nabla_\theta\log\pi_\theta(A_t|S_t)\hat A_t.

\]

The baseline reduces variance but does not remove all sampling noise. Entropy regularization may encourage exploration:

\[

J_{ent}=J+\beta\mathbb{E}_\pi[\mathcal{H}(\pi(\cdot|S))],

\]

where \(\beta\geq0\) controls the strength of the regularizer.

Actor–critic architecture

An actor represents \(\pi_\theta(a|s)\), while a critic estimates \(V_w(s)\) or \(Q_w(s,a)\). A practical one-step algorithm is:

1. Observe \(s_t\), sample \(a_t\sim\pi_\theta(\cdot|s_t)\), and obtain \(r_{t+1},s_{t+1}\).

2. Form the target \(y_t=r_{t+1}+\gamma(1-d_t)V_w(s_{t+1})\), where \(d_t=1\) if the transition is terminal.

3. Compute \(\delta_t=y_t-V_w(s_t)\).

4. Update the critic by minimizing \(\frac12(y_t-V_w(s_t))^2\).

5. Update the actor using \(\theta\leftarrow\theta+\alpha_\theta\gamma^t\delta_t\nabla_\theta\log\pi_\theta(a_t|s_t)\).

For longer horizons, generalized advantage estimation (GAE) uses

\[

\hat A_t^{GAE(\gamma,\lambda)}=\sum_{l=0}^{T-t-1}(\gamma\lambda)^l\delta_{t+l}.

\]

The parameter \(\lambda\) controls the bias–variance trade-off. In implementations, advantage estimates are often standardized within a batch, while value targets may be clipped or normalized. These operations change optimization behavior and must be documented.

Modern actor–critic methods include trust-region or clipped objectives. Proximal Policy Optimization (PPO), for example, uses the probability ratio

\[

r_t(\theta)=\frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{old}}(a_t|s_t)}

\]

and the clipped surrogate

\[

L^{CLIP}=\mathbb{E}[\min(r_t\hat A_t,\operatorname{clip}(r_t,1-\epsilon,1+\epsilon)\hat A_t)].

\]

Clipping limits policy movement but does not provide an absolute guarantee of safety or monotonic improvement. Separate controls are needed for constraints and deployment risk.

For continuous actions, deterministic policy-gradient methods learn \(a=\mu_\theta(s)\). The actor gradient is approximately

\[

\nabla_\theta J\approx\mathbb{E}_{s\sim\mathcal D}[\nabla_a Q_w(s,a)|_{a=\mu_\theta(s)}\nabla_\theta\mu_\theta(s)].

\]

Algorithms such as TD3 use twin critics, delayed actor updates, and target-policy smoothing to reduce overestimation and instability. Soft Actor–Critic adds an entropy term and learns a stochastic policy, often improving robustness in continuous control.

5. Offline Reinforcement Learning

Offline, or batch, RL learns from a fixed dataset

\[

\mathcal D=\{(s_i,a_i,r_i,s'_i,d_i)\}_{i=1}^N

\]

without new environment interaction during training. This is important when exploration is expensive, unsafe, regulated, or impossible, as in healthcare, industrial plants, and historical IT operations.

The central difficulty is distribution shift. The learned policy may select actions rarely or never represented in the dataset. A critic trained using bootstrapping may assign unjustifiably high values to such out-of-distribution actions; this is called extrapolation error. Offline evaluation is also difficult because the true counterfactual outcome of an unobserved action is unavailable.

Formal fitted Q-learning procedure

A basic fitted Q iteration minimizes

\[

L_Q(w)=\frac{1}{N}\sum_{i=1}^N\left[Q_w(s_i,a_i)-y_i\right]^2,

\]

where

\[

y_i=r_i+\gamma(1-d_i)\max_{a'}Q_{w^-}(s'_i,a').

\]

Here \(w^-\) is a delayed target-network parameter. In offline use, the maximization can query actions outside the data distribution, causing overestimation. A conservative alternative penalizes high values for unsupported actions. A representative conservative Q-learning objective is

\[

L_{CQL}(w)=L_Q(w)+\alpha\left[\mathbb{E}_{s\sim\mathcal D}\log\sum_a\exp Q_w(s,a)-\mathbb{E}_{(s,a)\sim\mathcal D}Q_w(s,a)\right]

\]

for discrete actions. The first term raises a soft maximum over actions and the second preserves values of observed actions, thereby discouraging unsupported action values. For continuous actions, the log-sum-exp term is approximated using sampled actions from a policy, a prior distribution, or perturbations.

Behavior-regularized methods constrain the learned policy toward the behavior policy \(\pi_\beta\), for example by penalizing divergence:

\[

\max_\pi\;\mathbb{E}_{s\sim\mathcal D,a\sim\pi}[Q(s,a)]-\alpha D(\pi(\cdot|s)\|\pi_\beta(\cdot|s)).

\]

If behavior probabilities are known or estimated, importance sampling can correct distribution differences, but trajectory-wise ratios often have very high variance:

\[

\rho_{t:T}=\prod_{k=t}^{T}\frac{\pi(A_k|S_k)}{\pi_\beta(A_k|S_k)}.

\]

Reliable support overlap is therefore essential. If \(\pi(a|s)>0\) while \(\pi_\beta(a|s)=0\), the ratio is undefined and the action has no empirical basis in the dataset.

Practical offline RL workflow

A defensible workflow includes:

1. Validate state construction, timestamps, action units, missing values, and terminal indicators.

2. Split data by time or by operational unit, rather than randomly when temporal leakage is possible.

3. Estimate action coverage and identify states with weak support.

4. Train a behavior-cloning baseline and compare the RL policy against it.

5. Use conservative critics, target networks, regularization, and early stopping.

6. Evaluate with off-policy estimators, model-based stress tests, uncertainty intervals, and limited controlled trials.

7. Deploy gradually with action constraints, rollback mechanisms, monitoring, and human approval for high-risk actions.

Offline RL should not be presented as a license to optimize arbitrary historical logs. Logged rewards may reflect biased operators, changing policies, missing confounders, or incentives that are no longer valid.

6. Applied Case: Data-Centre Resource Allocation

The following is an illustrative case rather than a report of a particular organization.

Background and problem

A data centre must allocate compute capacity among interactive services and batch workloads. Demand changes over time. Excess capacity increases energy cost, while insufficient capacity causes latency violations and service-level penalties. Historical logs contain utilization, queue length, temperature, available servers, submitted jobs, chosen allocation, and observed cost.

MDP formulation

A state may be represented as

\[

s_t=(u_t,q_t,e_t,\ell_t,h_t),

\]

where \(u_t\) is utilization, \(q_t\) queue length, \(e_t\) energy price, \(\ell_t\) latency statistics, and \(h_t\) time or calendar information. The action is the number of servers assigned to each workload class, subject to capacity constraints. A reward can be defined as

\[

r_t=-(c_E E_t+c_L L_t+c_V V_t),

\]

where \(E_t\) is energy use, \(L_t\) latency, \(V_t\) an SLA violation measure, and the coefficients convert each quantity into a common cost unit. They must be selected with operations and governance stakeholders; arbitrary reward weights can create undesirable priorities.

Analysis and solution

If a reliable simulator is available, model-based planning or online actor–critic training can be tested in simulation. Because exploration on production workloads is risky, an offline approach is initially preferable. The team should first train behavior cloning to reproduce historical allocation decisions, then train a conservative Q-function using logged transitions. The policy should be restricted to actions within observed capacity ranges and evaluated against behavior cloning, a threshold heuristic, and an operations-research scheduler.

A doubly robust off-policy estimate can combine a model-based value estimate with importance-weighted corrections, but confidence intervals are needed because estimates can be unstable under weak overlap. A shadow deployment can generate proposed actions without applying them. Only after stable shadow results should a constrained controller be introduced with a fallback scheduler.

Outcome and learning points

A successful outcome is not merely a higher simulated reward. It includes reduced SLA violations, bounded energy cost, stable behavior under demand spikes, and auditable decisions. The case demonstrates that state definition, reward governance, support coverage, baseline comparison, and safe deployment are as important as neural-network architecture.

7. Evaluation, Stability, and Engineering Practice

RL experiments are highly sensitive to random seeds, reward scaling, termination handling, exploration schedules, network initialization, and data preprocessing. A credible evaluation should report multiple seeds, confidence intervals or dispersion measures, learning curves, and performance on temporally separated scenarios. For control applications, include constraint violations, peak error, settling behavior, energy use, and worst-case or tail outcomes rather than only mean reward.

Common implementation errors include bootstrapping through terminal states, confusing truncation with termination, using future information in state features, incorrectly scaling actions, and evaluating an exploratory policy when the deployed policy is deterministic. Replay buffers should preserve enough information to reconstruct transitions and policy versions. Target networks should be updated using a documented hard or soft schedule.

In safety-critical systems, RL should usually be subordinate to a safety layer. Action shielding, control-barrier constraints, rule-based overrides, conservative action sets, and human escalation can reduce risk. These mechanisms may alter the effective policy and therefore must be included in evaluation.

8. Ethical, Legal, and Governance Issues

RL systems can optimize measurable outcomes while harming unmeasured stakeholders. Resource allocation may disadvantage particular users, locations, or service classes if historical rewards encode inequity. Audit logs should record state inputs, chosen actions, policy version, reward components, overrides, and uncertainty indicators. Personal or operational data must be handled under applicable privacy and cybersecurity requirements.

Organizations should define responsibility for reward design, model approval, incident response, and retirement. Offline datasets require provenance and access controls. A model should not be deployed solely because it outperforms a baseline in a simulator; simulation gaps and counterfactual uncertainty must be explicitly assessed.

9. Contemporary Developments and Research Directions

Current research includes distributional RL, which models a return distribution rather than only its mean; risk-sensitive and distributionally robust RL; multi-agent learning; model-based planning with learned world models; hierarchical RL; and offline-to-online adaptation. Decision transformers and related sequence-modeling methods treat trajectories as conditional sequence prediction, although their reliability remains dependent on data coverage and evaluation quality.

Important open problems include safe exploration, causal representation of state, non-stationary environments, partial observability, uncertainty calibration, long-horizon credit assignment, and principled offline policy evaluation. At postgraduate level, these are not merely algorithm-selection issues: they concern identifiability, statistical efficiency, computational scalability, and the validity of deployment claims.

Chapter Summary

Reinforcement learning formalizes sequential decision-making through MDPs, policies, transition dynamics, rewards, and discounted returns. Bellman equations support dynamic programming when a model is known. Monte Carlo, TD, SARSA, and Q-learning learn from sampled experience, with different bias, variance, and on/off-policy properties. Policy gradients optimize parameterized policies directly; actor–critic methods combine policy updates with learned value estimates and are suitable for large or continuous action spaces. Correct objective conventions matter: for \(J=\mathbb{E}[G_0]\), the REINFORCE trajectory estimator includes \(\gamma^t\) under the stated return formulation.

Offline RL avoids risky exploration but introduces distribution shift, unsupported actions, extrapolation error, and difficult counterfactual evaluation. Conservative critics, behavior regularization, coverage analysis, uncertainty estimates, and staged deployment are essential. Practical RL requires careful state and reward design, robust baselines, constraint handling, reproducible evaluation, governance, and safety mechanisms. The most effective engineering solution is therefore not necessarily the most sophisticated algorithm, but the one whose assumptions, data support, risks, and operational controls are demonstrably appropriate.

Key Terms

Markov decision process: A model in which the future depends on the current state and action, not the full history.

State: Information used to represent the decision situation at a time step.

Action: A choice made by the agent.

Policy: A rule or probability distribution mapping states to actions.

Reward: A scalar feedback signal measuring immediate desirability.

Return: The accumulated, usually discounted, future reward.

Discount factor: The parameter \(\gamma\) weighting future rewards.

Value function: Expected return from a state under a policy.

Action-value function: Expected return from a state–action pair.

Bellman equation: A recursive relationship expressing value through immediate reward and successor value.

Dynamic programming: Model-based solution of sequential decision problems using recursive value relationships.

Temporal-difference learning: Learning from a target combining observed reward with a bootstrapped estimate.

On-policy learning: Learning about the policy used to generate behavior.

Off-policy learning: Learning about a target policy from behavior generated by another policy.

Policy gradient: The gradient of expected performance with respect to policy parameters.

Actor–critic: An architecture in which an actor selects actions and a critic estimates value.

Advantage: The value of an action relative to a state baseline, commonly \(A(s,a)=Q(s,a)-V(s)\).

Offline RL: Learning a policy from a fixed dataset without additional interaction during training.

Extrapolation error: Incorrect value estimation for actions poorly represented in offline data.

Behavior policy: The policy that generated an observed dataset.

Review Questions

Short Answer Questions

1. What is the Markov property, and why is it important in RL?

2. Distinguish state value from action value.

3. What is the difference between SARSA and Q-learning targets?

4. Why does a baseline reduce policy-gradient variance without changing its expectation?

5. Define extrapolation error in offline reinforcement learning.

Long Answer Questions

1. Derive the Bellman expectation and Bellman optimality equations and explain their assumptions.

2. Compare Monte Carlo, TD(0), SARSA, and Q-learning in terms of bootstrapping, policy dependence, and variance.

3. Explain actor–critic learning, including critic targets, advantage estimation, and policy updates.

4. Analyze why offline RL is vulnerable to distribution shift and explain two mitigation strategies.

5. Design an evaluation and governance framework for an RL-based industrial controller.

Analytical / Application Questions

1. A system has \(\gamma=0.9\), immediate reward 3, and expected next-state value 10.5111. Calculate the action value and interpret its meaning.

2. Design an MDP for allocating cloud servers among interactive and batch workloads. Specify state variables, actions, rewards, constraints, and one potential partial-observability problem.

3. A historical dataset contains no examples of a proposed action in a frequently visited state. Explain why a learned offline critic may assign an unreliable value and recommend a deployment response.

Multiple Choice Questions with Answers

Question 1

Which equation defines the discounted return from time t?

A. G_t = R_t + γR_{t-1} + γ²R_{t-2}

B. G_t = Σ_{k=0}∞ γ^k R_{t+k+1}

C. G_t = Σ_{k=1}∞ R_{t+k}/γ^k

D. G_t = γR_{t+1} only

Correct Answer: B. G_t = Σ_{k=0}∞ γ^k R_{t+k+1}

Question 2

Which condition is required for the standard MDP Markov property?

A. The reward must always be positive

B. The action space must be continuous

C. The future distribution depends on the current state and action, not the full history

D. The policy must be deterministic

Correct Answer: C. The future distribution depends on the current state and action, not the full history

Question 3

With γ = 0.9, immediate reward 3, and expected next-state value 10.5111, what is Q?

A. 9.4600

B. 12.4600

C. 13.5111

D. 31.5333

Correct Answer: B. 12.4600

Question 4

Which method uses the next action actually selected by the behavior policy in its TD target?

A. SARSA

B. Q-learning

C. Value iteration

D. Behavior cloning

Correct Answer: A. SARSA

Question 5

For J = E[G0] with the stated return-based episodic convention, which factor appears in the REINFORCE sum?

A. γ^t

B. 1/γ^t only

C. γ^{-T}

D. No time-dependent factor can ever appear

Correct Answer: A. γ^t

Question 6

What is the primary role of a critic in an actor–critic algorithm?

A. To encrypt the replay buffer

B. To estimate values or advantages used to guide the actor

C. To guarantee constraint satisfaction

D. To replace the environment

Correct Answer: B. To estimate values or advantages used to guide the actor

Question 7

Why is offline RL especially vulnerable to extrapolation error?

A. It always uses an excessively small discount factor

B. It cannot use rewards

C. The learned policy may select actions poorly represented in the dataset

D. It requires every action to be deterministic

Correct Answer: C. The learned policy may select actions poorly represented in the dataset

Question 8

In fitted Q-learning, what is the purpose of a target network?

A. To make the environment stochastic

B. To provide a more slowly changing bootstrap target

C. To remove the reward signal

D. To guarantee unbiased policy evaluation

Correct Answer: B. To provide a more slowly changing bootstrap target

Question 9

Which approach directly encourages an offline policy to remain near the behavior policy?

A. Behavior regularization using a divergence penalty

B. Increasing the action space without data

C. Removing terminal indicators

D. Maximizing unsupported critic values

Correct Answer: A. Behavior regularization using a divergence penalty

Question 10

Which deployment practice is most appropriate for a high-risk RL controller?

A. Immediate unrestricted deployment after one simulation run

B. Disable monitoring to prevent feedback effects

C. Use shadow testing, safety constraints, monitoring, and rollback mechanisms

D. Evaluate only average reward

Correct Answer: C. Use shadow testing, safety constraints, monitoring, and rollback mechanisms