Policy Gradient and Actor–Critic Methods

Introduction

Value-based reinforcement learning estimates how desirable states or actions are and derives a policy from those estimates. Policy-gradient methods take a different approach: they parameterize the policy directly and adjust its parameters in the direction that increases expected return. This is especially useful when actions are continuous, policies must remain stochastic, or the action space is too large for explicit maximization over actions.

This chapter develops the policy-gradient theorem, the REINFORCE algorithm, variance-reduction baselines, advantage estimation, actor–critic architectures, entropy regularization, and continuous-action optimization. It also provides an executable PyTorch implementation for a discrete-action actor–critic agent.

Learning Objectives

By the end of this chapter, you should be able to:

1. Derive the likelihood-ratio policy-gradient estimator.

2. Explain the role of discounting and correctly implement the factor \(\gamma^t\).

3. Implement REINFORCE with returns and a baseline.

4. Distinguish rewards, returns, values, advantages, and temporal-difference errors.

5. Design an actor–critic learner using semi-gradient critic updates.

6. Apply entropy regularization to control exploration.

7. Optimize policies with continuous actions using Gaussian distributions.

8. Evaluate practical limitations, implementation choices, and ethical risks.

1. The Policy-Optimization Perspective

A Markov decision process consists of states \(s_t\), actions \(a_t\), transition probabilities \(P(s_{t+1}\mid s_t,a_t)\), reward \(r_t\), and discount factor \(\gamma\in[0,1)\). A stochastic policy \(\pi_\theta(a\mid s)\) assigns action probabilities or densities using parameters \(\theta\).

For an episodic task, define the return from time \(t\) as

\[

G_t=\sum_{k=0}^{T-1-t}\gamma^k r_{t+k},

\]

where \(T\) is the terminal time. The performance objective is

\[

J(\theta)=\mathbb E_{\tau\sim\pi_\theta}[G_0],

\]

where \(\tau=(s_0,a_0,r_0,\ldots,s_T)\) is a trajectory. The policy-gradient objective directly seeks parameters that maximize expected return:

\[

\theta\leftarrow\theta+\alpha_\theta\widehat{\nabla_\theta J}.

\]

Unlike Q-learning, policy optimization does not require a differentiable environment. The environment can be treated as a black box; differentiability is needed only for the policy and, when used, the critic.

2. Deriving the Policy Gradient

The probability of a trajectory is

\[

p_\theta(\tau)=\rho(s_0)\prod_{t=0}^{T-1}\pi_\theta(a_t\mid s_t)P(s_{t+1}\mid s_t,a_t),

\]

where \(\rho\) is the initial-state distribution. Assuming the transition dynamics do not depend on \(\theta\),

\[

\nabla_\theta J(\theta)=\nabla_\theta\sum_\tau p_\theta(\tau)G_0.

\]

Using \(\nabla p=p\nabla\log p\),

\[

\nabla_\theta J(\theta)=\mathbb E_\tau\left[G_0\nabla_\theta\log p_\theta(\tau)\right].

\]

Only the policy terms depend on \(\theta\), so

\[

\nabla_\theta\log p_\theta(\tau)=\sum_{t=0}^{T-1}\nabla_\theta\log\pi_\theta(a_t\mid s_t).

\]

Thus a basic estimator is

\[

\widehat g=G_0\sum_{t=0}^{T-1}\nabla_\theta\log\pi_\theta(a_t\mid s_t).

\]

A causality argument improves this estimator. An action at time \(t\) cannot affect rewards preceding time \(t\). Replacing the full return by the reward-to-go gives the commonly used discounted estimator

\[

\boxed{

\widehat{\nabla_\theta J}=

\sum_{t=0}^{T-1}\gamma^t\nabla_\theta\log\pi_\theta(a_t\mid s_t)G_t

}

\]

when \(J(\theta)=\mathbb E[G_0]\) and \(G_t\) is discounted from time \(t\). The factor \(\gamma^t\) is required under this time-origin convention. Some texts define an alternative objective based on a discounted occupancy measure; in that formulation the factor is absorbed into the state-visitation distribution. These conventions must not be mixed.

For continuing tasks, one generally uses an average-reward objective or a discounted objective with a suitable initial-state distribution and convergence conditions. The finite-episode expression above is directly applicable to episodic learning.

3. REINFORCE

REINFORCE is the Monte Carlo policy-gradient algorithm:

1. Generate an episode using the current policy.

2. Compute each \(G_t\).

3. Ascend the policy objective using \(\sum_t\gamma^t\log\pi_\theta(a_t\mid s_t)G_t\).

Because standard machine-learning optimizers minimize losses, define

\[

L_{\text{policy}}=-\sum_{t=0}^{T-1}\gamma^t\log\pi_\theta(a_t\mid s_t)G_t.

\]

Minimizing this loss performs gradient ascent on expected return. REINFORCE is unbiased under correct sampling, but it can have high variance because complete returns include random future rewards.

Worked numerical example

Suppose an episode has rewards \((2,0,3)\), \(\gamma=0.5\), and policy-log-gradient vectors

\(z_0=(1,0)\), \(z_1=(0,1)\), and \(z_2=(1,1)\), where \(z_t=\nabla\log\pi(a_t\mid s_t)\). The returns are

\[

G_0=2+0.5(0)+0.5^2(3)=2.75,

\quad G_1=0+0.5(3)=1.5,

\quad G_2=3.

\]

The discounted estimator is

\[

0.5^0(2.75)(1,0)+0.5^1(1.5)(0,1)+0.5^2(3)(1,1)

=(3.50,1.50).

\]

This is not the same as omitting the \(\gamma^t\) factors.

4. Baselines and Advantages

A baseline can reduce variance without changing the expected policy gradient if it does not depend on the sampled action. For any state-dependent \(b(s_t)\),

\[

\mathbb E_{a_t\sim\pi_\theta(\cdot\mid s_t)}

[\nabla_\theta\log\pi_\theta(a_t\mid s_t)b(s_t)]=0,

\]

because

\[

\sum_a\pi_\theta(a\mid s)\nabla\log\pi_\theta(a\mid s)=

\nabla\sum_a\pi_\theta(a\mid s)=0.

\]

For continuous actions, the sum becomes an integral under regularity conditions. The preferred baseline is often the state-value function \(V^\pi(s_t)\). Define the advantage

\[

A^\pi(s_t,a_t)=Q^\pi(s_t,a_t)-V^\pi(s_t).

\]

A Monte Carlo advantage estimate is

\[

\widehat A_t=G_t-\widehat V_\phi(s_t).

\]

The actor loss becomes

\[

L_{\text{actor}}=-\sum_t\gamma^t\log\pi_\theta(a_t\mid s_t)\operatorname{stopgrad}(\widehat A_t).

\]

Detaching the advantage prevents the actor update from unintentionally optimizing through the critic.

A learned baseline can be trained by regression:

\[

L_V=\frac12\sum_t(\widehat V_\phi(s_t)-\operatorname{stopgrad}(G_t))^2.

\]

Normalizing a batch of advantages, \((A-\bar A)/(\sigma_A+\epsilon)\), often improves numerical stability, although it changes the scale of the update and should be viewed as an optimization heuristic rather than a policy-gradient identity.

5. Actor–Critic Learning

REINFORCE waits until an episode ends. Actor–critic methods use a critic to estimate value during interaction. The actor represents \(\pi_\theta\); the critic represents \(V_\phi(s)\).

For a one-step transition, the TD error is

\[

\delta_t=r_t+\gamma(1-d_t)V_\phi(s_{t+1})-V_\phi(s_t),

\]

where \(d_t=1\) if the transition reaches a true terminal state and zero otherwise. If a time limit is merely a truncation imposed by the simulator, bootstrapping may still be appropriate; confusing truncation with termination produces biased targets.

The standard critic update uses the target

\[

y_t=r_t+\gamma(1-d_t)\operatorname{stopgrad}(V_\phi(s_{t+1})),

\]

and minimizes the semi-gradient loss

\[

L_V=\frac12\left(V_\phi(s_t)-y_t\right)^2.

\]

The bootstrap value is detached. Differentiating through \(V_\phi(s_{t+1})\) would not be the standard TD semi-gradient update and can create an unintended moving-target objective.

The actor can use \(\delta_t\) as an advantage estimate:

\[

L_{\text{actor}}=-\log\pi_\theta(a_t\mid s_t)\operatorname{stopgrad}(\delta_t).

\]

For episodic discounted objectives, an implementation using time-indexed trajectory losses should retain the appropriate \(\gamma^t\) weighting. In many continuing actor–critic implementations, the objective is defined using a discounted occupancy distribution, so the weighting is represented implicitly by the distribution of visited states. The objective convention should be documented.

n-step returns and generalized advantage estimation

An n-step return is

\[

\widehat G_t^{(n)}=\sum_{k=0}^{n-1}\gamma^k r_{t+k}+\gamma^nV(s_{t+n}),

\]

with the final bootstrap removed when the episode terminates. Larger \(n\) reduces bootstrapping bias but increases variance.

Generalized Advantage Estimation (GAE) combines TD errors:

\[

\widehat A_t^{\text{GAE}(\gamma,\lambda)}=

\sum_{l=0}^{T-1-t}(\gamma\lambda)^l\delta_{t+l},

\quad 0\leq\lambda\leq1.

\]

Here \(\lambda=0\) uses one-step TD errors and \(\lambda=1\) approaches a Monte Carlo-style estimate, subject to terminal handling and the quality of the critic.

6. Entropy Regularization

A stochastic policy has entropy

\[

\mathcal H(\pi(\cdot\mid s))=-\sum_a\pi(a\mid s)\log\pi(a\mid s).

\]

A regularized objective is

\[

J_{\text{reg}}=J+\beta\mathbb E[\mathcal H(\pi(\cdot\mid s))],

\quad \beta\geq0.

\]

Therefore the minimization loss is commonly written

\[

L=L_{\text{actor}}+c_VL_V-\beta\mathcal H.

\]

Entropy encourages exploration and prevents premature collapse to deterministic actions. Excessive \(\beta\), however, can maintain random behavior after exploration is no longer useful. A schedule that decreases \(\beta\), or automatic temperature tuning, may be appropriate.

7. Continuous-Action Policies

For continuous control, a common policy is a diagonal Gaussian:

\[

a\sim\mathcal N(\mu_\theta(s),\operatorname{diag}(\sigma_\theta(s)^2)),

\quad \sigma_i=\exp(\ell_i),

\]

where \(\ell_i\) is a learned log standard deviation. Its log density is

\[

\log\pi(a\mid s)=

-\frac12\sum_i\left[

\left(\frac{a_i-\mu_i}{\sigma_i}\right)^2+2\log\sigma_i+\log(2\pi)

\right].

\]

The likelihood-ratio policy gradient remains

\[

\nabla_\theta J=\mathbb E[\nabla_\theta\log\pi_\theta(a\mid s)A(s,a)].

\]

When using this estimator, the sampled action is treated as fixed (detached) while evaluating \(\log\pi_\theta(a\mid s)\). If an implementation allows an accidental path through a reparameterized sample \(a=\mu+\sigma\epsilon\), it can mix pathwise and likelihood-ratio derivatives and produce an incorrect estimator. A reparameterized gradient is valid only when the objective and derivation explicitly support it; it is not a harmless replacement inside a score-function loss.

Actions are often squashed to bounds using \(u\sim\mathcal N(\mu,\sigma^2)\), \(a=\tanh(u)\). Then the log density of the bounded action must include the change-of-variables correction:

\[

\log\pi(a\mid s)=\log\mathcal N(u;\mu,\sigma^2)-

\sum_i\log(1-\tanh(u_i)^2+\epsilon).

\]

Ignoring this correction gives an incorrect density and therefore an incorrect entropy and policy gradient. Environment action bounds, scaling, clipping, and numerical stability must also be handled explicitly.

8. Complete PyTorch Implementation: Discrete Actor–Critic

The following script is executable with PyTorch and Gymnasium installed. It uses a one-step actor–critic algorithm for CartPole. The target value is detached, the TD advantage is detached for the actor, and terminated and truncated transitions are handled separately. For this continuing-style online implementation, the objective is expressed through visited states rather than an explicitly time-indexed episodic \(\gamma^t\) sum. A full episodic REINFORCE implementation would multiply each time-step score term by \(\gamma^t\) as derived above.

```python

pip install torch gymnasium

import gymnasium as gym

import torch

import torch.nn as nn

from torch.distributions import Categorical

class ActorCritic(nn.Module):

def __init__(self, obs_dim, n_actions):

super().__init__()

self.body = nn.Sequential(

nn.Linear(obs_dim, 128), nn.Tanh(),

nn.Linear(128, 128), nn.Tanh())

self.policy_head = nn.Linear(128, n_actions)

self.value_head = nn.Linear(128, 1)

def forward(self, obs):

h = self.body(obs)

return self.policy_head(h), self.value_head(h).squeeze(-1)

def train(episodes=500, gamma=0.99, entropy_coef=0.01):

env = gym.make("CartPole-v1")

obs_dim = env.observation_space.shape[0]

n_actions = env.action_space.n

model = ActorCritic(obs_dim, n_actions)

optimizer = torch.optim.Adam(model.parameters(), lr=3e-4)

for ep in range(episodes):

obs, _ = env.reset()

done = False

total = 0.0

while not done:

state = torch.as_tensor(obs, dtype=torch.float32).unsqueeze(0)

logits, value = model(state)

dist = Categorical(logits=logits)

action = dist.sample()

next_obs, reward, terminated, truncated, _ = env.step(action.item())

done = terminated or truncated

with torch.no_grad():

next_state = torch.as_tensor(

next_obs, dtype=torch.float32).unsqueeze(0)

_, next_value = model(next_state)

Bootstrap across time-limit truncation, not true termination.

bootstrap = 0.0 if terminated else next_value.item()

target = torch.tensor(

reward + gamma * bootstrap, dtype=torch.float32)

Semi-gradient TD critic: target is detached by construction.

critic_loss = 0.5 * (value.squeeze(0) - target).pow(2)

advantage = (target - value.squeeze(0)).detach()

actor_loss = -dist.log_prob(action).squeeze(0) * advantage

entropy_loss = -entropy_coef * dist.entropy().squeeze(0)

loss = actor_loss + critic_loss + entropy_loss

optimizer.zero_grad()

loss.backward()

nn.utils.clip_grad_norm_(model.parameters(), 0.5)

optimizer.step()

obs = next_obs

total += reward

if (ep + 1) % 25 == 0:

print(f"episode={ep+1}, return={total:.1f}")

env.close()

if __name__ == "__main__":

train()

```

This is a baseline implementation, not a guarantee of a particular score. Reproducible evaluation should use fixed seeds, multiple independent runs, confidence intervals, and a separate evaluation policy with exploration controlled or disabled.

9. Applied Case: Illustrative Warehouse-Robot Navigation

Background

A warehouse robot must move shelves or packages while avoiding collisions and excessive energy use. The state may include lidar features, robot velocity, goal direction, and battery level. Actions may be discrete motion commands or continuous linear and angular velocities.

Problem

A value table is infeasible because sensor observations are high-dimensional and velocity commands are continuous. Pure REINFORCE can learn but may be unstable because collisions are sparse and delayed.

Approach

An actor outputs a Gaussian distribution over velocity commands. A critic estimates the state value. Rewards combine progress toward the goal, a collision penalty, energy cost, and a terminal success bonus. GAE reduces variance, entropy is gradually reduced, and actions are bounded through a correctly corrected tanh-Gaussian policy. Training occurs in simulation with randomized friction, sensor noise, layouts, and obstacle positions.

Outcome and evaluation

A credible evaluation compares the learned policy with a hand-designed controller and reports success rate, collision rate, path length, energy use, and worst-case behavior under distribution shifts. A high mean return alone is insufficient: a policy that occasionally causes dangerous collisions may be unacceptable.

Learning points

The critic improves sample efficiency but can introduce bias. Reward design is a safety specification, not merely a numerical convenience. Simulation-to-real transfer requires domain randomization, conservative deployment, monitoring, a fail-safe controller, and a human override.

10. Limitations, Governance, and Best Practices

Policy gradients can suffer from high variance, local optima, unstable value estimates, sensitivity to learning rates, and poor credit assignment over long horizons. Neural policies may also exploit reward loopholes, such as repeatedly triggering a poorly designed reward event.

Best practices include:

• Specify the objective convention and discounting convention explicitly.
• Use detached returns, targets, and advantages where required.
• Distinguish termination from time-limit truncation.
• Normalize observations and inspect reward scales.
• Clip gradients and monitor entropy, value loss, explained variance, and approximate KL divergence.
• Evaluate across seeds and unseen conditions.
• Check action bounds and continuous-density corrections.
• Keep safety constraints separate from soft reward preferences when failure has serious consequences.
• Log policy distributions, not only episode returns.

In consequential applications, fairness, privacy, accountability, and contestability matter. A policy optimizing engagement, for example, may learn manipulative behavior even if its numerical reward improves. Governance should define prohibited actions, audit logs, intervention authority, rollback procedures, and criteria for human approval.

11. Contemporary Developments

Modern policy optimization often uses clipped or trust-region updates, such as PPO and TRPO, to prevent excessively large policy changes. Off-policy actor–critic methods improve sample reuse but require corrections or objectives designed for replay data. Distributional critics, recurrent policies, multi-agent methods, constrained reinforcement learning, and offline reinforcement learning extend the framework but introduce additional assumptions. Offline policy optimization is particularly sensitive to distribution shift because the learner cannot freely collect corrective experience.

Chapter Summary

Policy-gradient methods optimize a parameterized policy directly using the likelihood-ratio identity. For the episodic objective \(J(\theta)=\mathbb E[G_0]\), with return \(G_t\) discounted from time \(t\), the trajectory estimator is \(\sum_t\gamma^t\nabla\log\pi(a_t\mid s_t)G_t\). REINFORCE is simple and unbiased in expectation but often has high variance. State-dependent baselines preserve unbiasedness and produce advantage estimates. Actor–critic methods replace complete Monte Carlo returns with bootstrapped value estimates; their critic normally uses a detached bootstrap target and a semi-gradient update. GAE provides a tunable bias–variance compromise. Entropy regularization supports exploration. Continuous-action policies commonly use Gaussian distributions, require careful action-bounding corrections, and must treat sampled actions as fixed when applying score-function gradients. Successful deployment requires sound reward design, robust evaluation, safety controls, and explicit governance.

Key Terms

Policy gradient: Gradient of expected return with respect to policy parameters.

Likelihood-ratio estimator: Identity expressing a gradient through \(\nabla\log p\).

REINFORCE: Monte Carlo policy-gradient algorithm using sampled returns.

Return: Discounted sum of future rewards.

Reward-to-go: Return beginning at the current time step.

Baseline: Action-independent value subtracted to reduce gradient variance.

Value function: Expected return from a state under a policy.

Action-value function: Expected return from a state–action pair.

Advantage: Action value minus state value.

Actor: Policy model being optimized.

Critic: Model estimating value or advantage information.

TD error: Difference between a bootstrapped target and current value estimate.

Semi-gradient: Update that treats the bootstrap target as fixed.

GAE: Generalized Advantage Estimation using weighted TD errors.

Entropy regularization: Objective term encouraging stochastic exploration.

Score function: Gradient of a log probability or density.

Reparameterization: Expressing a random sample as a deterministic transform of noise.

Truncation: Artificial episode ending, such as a time limit.

Termination: Environment-defined absorbing end of an episode.

Policy collapse: Loss of stochastic diversity before adequate exploration.

Review Questions

Short Answer Questions

1. Why is a policy-gradient method able to learn without differentiating through the environment?

2. State the episodic discounted REINFORCE estimator for \(J(\theta)=\mathbb E[G_0]\).

3. Why can a state-dependent baseline reduce variance without changing the expected gradient?

4. What is the difference between a TD target and a TD error?

5. Why is entropy included with a negative sign in a loss that is minimized?

Long Answer Questions

1. Derive the likelihood-ratio policy-gradient identity from the trajectory probability.

2. Compare REINFORCE and one-step actor–critic methods in terms of bias, variance, and sample efficiency.

3. Explain GAE and the role of \(\lambda\) in the bias–variance trade-off.

4. Derive the bounded tanh-Gaussian log-density correction and explain why it is needed.

5. Discuss reward design, safety, and governance for a reinforcement-learning robot.

Analytical / Application Questions

1. For rewards \((1,2)\), \(\gamma=0.9\), and score vectors \(z_0=(1,0)\), \(z_1=(0,1)\), calculate the discounted reward-to-go policy-gradient estimate.

2. A critic estimates \(V(s_t)=4\), receives reward \(r_t=-1\), estimates \(V(s_{t+1})=5\), uses \(\gamma=0.9\), and the transition is nonterminal. Calculate the TD target and TD error.

3. Design an evaluation protocol for a continuous-action warehouse robot that measures performance, safety, robustness, and reproducibility rather than relying only on mean return.

Multiple Choice Questions with Answers

Question 1

For the episodic objective J(θ)=E[G₀], where Gₜ is discounted from time t, which estimator is appropriate?

A. Σₜ ∇ log π(aₜ|sₜ)Gₜ

B. Σₜ γᵗ∇ log π(aₜ|sₜ)Gₜ

C. Σₜ γ⁻ᵗ∇ log π(aₜ|sₜ)Gₜ

D. Σₜ γᵗ log π(aₜ|sₜ) without a return

Correct Answer: B. Σₜ γᵗ∇ log π(aₜ|sₜ)Gₜ

Question 2

Which quantity is normally detached in a standard one-step TD critic target?

A. The immediate reward

B. The current state representation

C. The bootstrap value V(sₜ₊₁) inside the target

D. The environment action

Correct Answer: C. The bootstrap value V(sₜ₊₁) inside the target

Question 3

Which baseline preserves the expected policy gradient?

A. A baseline depending only on the current state

B. A baseline chosen after observing the sampled action

C. A baseline equal to the policy probability

D. A baseline depending on future random actions

Correct Answer: A. A baseline depending only on the current state

Question 4

If rewards are (2, 0, 3) and γ=0.5, what is G₁?

A. 1.5

B. 2.5

C. 3.0

D. 0.5

Correct Answer: A. 1.5

Question 5

What does λ=0 in GAE most closely produce?

A. A full undiscounted episode return

B. A one-step TD-error estimate

C. An entropy-only objective

D. A deterministic policy

Correct Answer: B. A one-step TD-error estimate

Question 6

What is the principal purpose of entropy regularization?

A. To force all rewards to be positive

B. To encourage exploration and prevent premature policy collapse

C. To remove the critic from the algorithm

D. To make the environment differentiable

Correct Answer: B. To encourage exploration and prevent premature policy collapse

Question 7

For a Gaussian likelihood-ratio policy gradient, how should a sampled action be treated when evaluating log probability?

A. As a detached fixed sample

B. As a target requiring no probability density

C. As an action with zero variance

D. As a value-function parameter

Correct Answer: A. As a detached fixed sample

Question 8

Why is a Jacobian correction needed for a tanh-squashed Gaussian policy?

A. Tanh changes the action density under a nonlinear transformation

B. Tanh makes rewards undiscounted

C. The critic cannot process continuous states

D. The policy becomes deterministic immediately

Correct Answer: A. Tanh changes the action density under a nonlinear transformation

Question 9

With V(sₜ)=4, rₜ=-1, γ=0.9, V(sₜ₊₁)=5, and a nonterminal transition, what is the TD error?

A. 0.5

B. -0.5

C. 4.5

D. -1.0

Correct Answer: B. -0.5

Question 10

Which statement best distinguishes truncation from termination?

A. Truncation always means the state is absorbing

B. Termination is an artificial time limit, while truncation is environmental

C. Truncation may be an artificial limit and can permit bootstrapping

D. They are mathematically identical in every implementation

Correct Answer: C. Truncation may be an artificial limit and can permit bootstrapping