Deep Reinforcement Learning

Introduction and Context

Deep reinforcement learning (deep RL) combines reinforcement learning (RL) with neural-network function approximation. Classical tabular RL stores a value for every state or state–action pair; this becomes infeasible when observations are images, sensor streams, language, or high-dimensional continuous measurements. Deep RL replaces tables with learned functions that generalize across related situations.

An RL agent interacts with an environment. At time t, it observes state s_t, selects action a_t according to policy π, receives reward r_{t+1}, and moves to s_{t+1}. The objective is to maximize expected discounted return:

G_t = r_{t+1} + γr_{t+2} + γ²r_{t+3} + …

where γ ∈ [0,1) is the discount factor. The action-value function is Q^π(s,a) = E_π[G_t | s_t=s, a_t=a], and the optimal value function satisfies the Bellman optimality equation:

Q(s,a) = E[r_{t+1} + γ max_{a'} Q(s_{t+1},a') | s_t=s,a_t=a].

The expectation is over environmental transitions. In a finite episodic problem, γ may equal 1 if returns remain well defined; continuing tasks generally use γ < 1.

Deep RL is powerful but difficult to train. Neural networks introduce approximation error, non-stationary targets, correlated data, unstable bootstrapping, and sensitivity to scale and hyperparameters. This chapter develops the principal deep RL algorithms used for discrete and continuous control, then examines stability, sample efficiency, evaluation, and responsible deployment.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain why neural networks are needed in large or continuous RL problems.

2. Construct and analyze DQN systems using replay buffers and target networks.

3. Distinguish value-based, policy-gradient, actor–critic, and maximum-entropy methods.

4. Explain the designs and trade-offs of DQN extensions, PPO, DDPG, TD3, and SAC.

5. Diagnose instability, poor sample efficiency, and hyperparameter sensitivity.

6. Design an evaluation and deployment plan for a practical deep RL application.

1. From Tabular RL to Neural Function Approximation

In Q-learning, a tabular update for a transition (s,a,r,s') is

Q(s,a) ← Q(s,a) + α[r + γ max_{a'}Q(s',a') − Q(s,a)],

where α is the learning rate. With a neural network Q_θ(s,a), the temporal-difference (TD) error is

δ = y − Q_θ(s,a),

and a common target is y = r + γ max_{a'}Q_θ(s',a') for nonterminal transitions. The loss is

L(θ) = E[(y − Q_θ(s,a))²].

The expectation is estimated from sampled transitions. If s' is terminal, the bootstrap term is omitted, so y=r. Neural approximation permits generalization, but the target depends on the same parameters being updated. This feedback can cause divergence, especially when data are correlated and the target changes rapidly.

Three design principles address this problem:

• Experience replay stores past transitions and samples randomized mini-batches.
• A target network supplies a slowly changing estimate of the bootstrap target.
• Careful exploration, normalization, optimization, and evaluation reduce variance and bias.

Important terminology includes on-policy learning, in which updates use data generated by the current policy, and off-policy learning, in which data may come from another behavior policy. DQN, DDPG, TD3, and SAC are primarily off-policy; PPO is generally treated as an on-policy method.

2. Deep Q-Networks

2.1 Basic DQN architecture

A Deep Q-Network approximates Q(s,a) with a neural network. For discrete actions, the network commonly receives a state and outputs one value for each action. The greedy action is

a_t = argmax_a Q_θ(s_t,a).

An ε-greedy behavior policy selects a random action with probability ε and a greedy action otherwise. ε is usually reduced during training, but it should not be reduced so aggressively that useful exploration disappears.

DQN stores transitions in a replay buffer D. A mini-batch of transitions is sampled uniformly, and the loss is computed using a target network with parameters θ⁻:

y_i = r_i + γ(1-d_i) max_{a'} Q_{θ⁻}(s'_i,a'),

where d_i is 1 if the transition ends an episode and 0 otherwise. The online parameters θ are optimized to minimize

L(θ) = (1/B) Σ_i [y_i − Q_θ(s_i,a_i)]²,

where B is the batch size. Every C environment steps, θ⁻ may be copied from θ. Alternatively, a soft update uses θ⁻ ← τθ + (1−τ)θ⁻, with small τ.

The target network is not intended to improve the final representation directly; its purpose is to reduce rapid target movement. Replay reduces temporal correlation and allows valuable experiences to be reused.

2.2 DQN training procedure

A practical algorithm is:

1. Initialize online network, target network, replay buffer, and exploration schedule.

2. Observe s_t and select an ε-greedy action.

3. Execute the action and store (s_t,a_t,r_{t+1},s_{t+1},d_t).

4. After a warm-up period, sample a mini-batch from the buffer.

5. Compute targets with the target network and optimize the online network.

6. Periodically update the target network.

7. Evaluate periodically without exploratory actions.

The warm-up period prevents early updates from being based on a tiny and unrepresentative data set. The replay capacity controls how much historical experience remains available. A very small buffer produces correlated data; an excessively large buffer can retain obsolete experiences after the policy or environment changes.

2.3 Worked example: one DQN target

Suppose a nonterminal transition has reward r=2, discount γ=0.9, and target-network action values at the next state [3.0, 1.5, 2.2]. The maximum next value is 3.0. Therefore,

y = 2 + 0.9(3.0) = 4.7.

If the online network predicts Q(s,a)=4.1, the TD error is 0.6 and the squared loss for this individual transition is 0.36. This target is a training target, not a guaranteed estimate of the actual return; it contains approximation and sampling error.

3. Important DQN Extensions

Double DQN

Standard DQN uses the target network both to select and evaluate the next action. Maximization over noisy estimates tends to create positive bias. Double DQN separates selection from evaluation:

y = r + γ(1-d)Q_{θ⁻}(s', argmax_{a'}Q_θ(s',a')).

The online network selects the action, while the target network evaluates it. This often reduces overestimation without requiring a second independently trained online network.

Dueling networks

A dueling network decomposes the value of a state and the relative advantage of actions. A common aggregation is

Q(s,a) = V(s) + A(s,a) − (1/|A|)Σ_{a'}A(s,a').

Here V(s) estimates the state value and A(s,a) measures the action’s relative usefulness. The subtraction fixes an identifiability problem: without a constraint, many combinations of V and A produce the same Q. Dueling architectures can be useful when many actions have similar consequences.

Prioritized experience replay

Uniform replay samples all stored transitions with equal probability. Prioritized replay samples transitions according to an estimate of learning significance, often related to |δ|. Importance-sampling weights partially correct the resulting distribution shift. Prioritization can accelerate learning but may over-sample noisy or surprising transitions and adds parameters such as priority exponent and bias-correction schedule.

Distributional and multi-step methods

Distributional RL estimates a distribution of possible returns rather than only their mean. This can provide richer learning signals and support risk-sensitive decisions, although the distributional target must be defined consistently with the chosen algorithm. Multi-step returns use several rewards before bootstrapping, for example

G_t^{(n)} = r_{t+1} + γr_{t+2} + … + γ^{n-1}r_{t+n} + γ^n V(s_{t+n}),

when the episode has not terminated earlier. Larger n can propagate rewards faster but may increase variance and sensitivity to inaccurate trajectories.

4. Policy-Gradient and Actor–Critic Methods

Value-based methods are inconvenient for continuous actions because maximizing Q(s,a) over a continuous action space may itself require difficult optimization. Policy-gradient methods instead learn a policy directly.

For a stochastic policy π_θ(a|s), the policy-gradient objective is J(θ)=E[return]. The policy-gradient theorem gives, under standard regularity conditions,

∇_θJ(θ) = E[∇_θ log π_θ(a|s) Q^π(s,a)].

A baseline that does not depend on the selected action can reduce variance. Using a value estimate V_φ(s), the advantage is

A^π(s,a)=Q^π(s,a)−V^π(s).

Actor–critic methods use an actor for the policy and a critic for value or action-value estimation. The critic can make updates more efficient, but errors in the critic can bias the actor.

5. Proximal Policy Optimization

PPO is an on-policy actor–critic algorithm designed to prevent excessively large policy updates. Let

r_t(θ)=π_θ(a_t|s_t)/π_{θ_old}(a_t|s_t)

be the probability ratio and Â_t an estimated advantage. The clipped surrogate objective is

L^{CLIP}(θ)=E[min(r_t(θ)Â_t, clip(r_t(θ),1−ε,1+ε)Â_t)].

The clipping parameter ε is not the exploration probability used in ε-greedy DQN. Clipping limits the incentive to move the new policy too far from the policy that generated the batch. PPO commonly adds a value-function loss and an entropy bonus:

L = policy loss + c_v(value loss) − c_e(entropy),

with signs chosen according to whether an implementation minimizes or maximizes the objective. Advantages are often standardized within a batch, which changes scale but not their ordering.

PPO discards or greatly reduces reuse of old data after several epochs, making it less sample efficient than replay-based off-policy algorithms but often relatively robust and simple to tune. It is effective for discrete or continuous actions, provided the policy distribution is parameterized appropriately—for example, categorical for discrete actions and a squashed Gaussian for bounded continuous actions.

6. Continuous-Control Algorithms

6.1 DDPG

Deep Deterministic Policy Gradient (DDPG) uses a deterministic actor μ_θ(s) and a critic Q_φ(s,a). Its critic target is

y = r + γ(1-d)Q_{φ⁻}(s', μ_{θ⁻}(s')).

The actor is updated to maximize Q_φ(s,μ_θ(s)), equivalently minimizing its negative. Target actors and critics are softly updated. Exploration is added to actions during data collection, commonly by noise, but the exact noise process should match the environment’s time scale and action constraints.

DDPG is off-policy and can be sample efficient, but it is sensitive to critic errors, action scaling, exploration noise, and hyperparameters. A deterministic policy may also explore poorly in multimodal or highly stochastic tasks.

6.2 TD3

Twin Delayed Deep Deterministic Policy Gradient (TD3) addresses several DDPG weaknesses:

1. Twin critics estimate Q_1 and Q_2; the smaller target is used:

y=r+γ(1-d)min_j Q_{j,φ⁻}(s',a').

2. Target policy smoothing adds clipped noise to the target action, reducing exploitation of narrow critic errors.

3. The actor is updated less frequently than the critics, allowing the critic to improve before it drives the policy.

TD3 is often a strong baseline for bounded continuous control, though it remains sensitive to reward scaling, action normalization, and exploration design.

6.3 Soft Actor-Critic

SAC optimizes expected reward together with policy entropy:

J(π)=E[Σ_t γ^t(r_{t+1}+α H(π(·|s_t)))],

where H is entropy and α>0 controls the reward–exploration trade-off. A common soft value target uses two critics:

y=r+γ(1-d)[min_j Q_{j,φ⁻}(s',a')−α log π_θ(a'|s')],

with a' sampled from the current policy. SAC generally uses stochastic actors and twin critics. Automatic entropy tuning can adjust α toward a target entropy, reducing one manual tuning burden. SAC is sample efficient and exploratory, but implementation details—action squashing, log-probability correction, reward scale, and termination handling—must be correct.

7. Stability, Sample Efficiency, and Hyperparameter Sensitivity

7.1 Sources of instability

Deep RL is a moving-target problem. Instability can result from:

• bootstrapping from inaccurate value estimates;
• the function approximator changing the target it is chasing;
• correlated sequential observations;
• distribution shift between data collection and learning;
• reward sparsity or extreme reward magnitudes;
• partial observability and insufficient state representation;
• nonstationary opponents or multi-agent environments;
• incorrect treatment of time limits as terminal failures.

A useful diagnostic process is to plot episodic returns, episode lengths, critic losses, TD-error distributions, action distributions, and evaluation performance over multiple random seeds. A decreasing loss does not prove that the policy is improving.

7.2 Sample efficiency

Sample efficiency concerns performance per environment interaction, not merely speed per gradient step. Replay, multi-step targets, model-based rollouts, demonstrations, and off-policy learning can improve reuse. However, more updates per transition can amplify errors and overfitting to the replay distribution. PPO often requires more environment samples but can be stable because it uses data close to the current policy.

Wall-clock efficiency is different: a method that uses fewer samples may still be slower if its networks or update schedule are computationally expensive. Both environment steps and computational cost should be reported.

7.3 Hyperparameters

Important choices include learning rates, batch size, discount factor, replay capacity, warm-up length, target-update rate, exploration schedule, network architecture, reward scale, gradient clipping, update-to-data ratio, entropy coefficient, and PPO clipping range. Hyperparameters interact: a larger learning rate may require slower target updates; a larger batch can reduce gradient variance but may reduce responsiveness; a high discount factor increases planning horizon and can worsen sensitivity to value errors.

Best practice is to define a validation protocol before tuning, use several random seeds, report mean and variation, and avoid selecting a method solely from its best run. Normalize observations where justified, scale or clip rewards cautiously, and preserve the semantics of terminal events.

8. Engineering a Deep RL System

A robust system begins with a precise Markov decision process. Specify observations, actions, transition timing, reward units, discount factor, terminal conditions, constraints, and safety requirements. The observation should contain sufficient information for good decisions; if not, frame stacking, recurrent networks, or belief-state methods may be needed.

For discrete actions, begin with DQN or Double DQN and add dueling or prioritized replay only when a clear need exists. For continuous actions, TD3 and SAC are practical baselines; DDPG is historically important but often less robust. PPO is attractive when on-policy stability, simple rollout logic, or policy constraints are more important than maximal sample reuse.

Use action normalization internally, then map normalized outputs to physical limits. Prevent invalid actions through environment-level constraints rather than relying only on penalties. Separate training, validation, and final test scenarios. In safety-critical systems, train in simulation, use conservative offline evaluation, impose a safety layer or fallback controller, and deploy gradually with monitoring.

9. Illustrative Case: Energy Storage for a Microgrid

Background

Consider an illustrative microgrid with a battery, photovoltaic generation, variable electricity prices, and a demand forecast. The controller must choose battery charge or discharge power every 15 minutes.

Problem

A continuous action is appropriate, but battery degradation, state-of-charge limits, forecast uncertainty, and peak-demand charges create competing objectives. A reward might combine energy cost, peak penalties, degradation estimates, and constraint violations. Reward components must be expressed in compatible units or explicitly weighted; otherwise the agent may optimize an unintended objective.

Analysis and approach

The state can include state of charge, recent load and solar observations, price, forecast features, time information, and battery temperature. A first baseline is a rule-based controller and a linear or model-predictive controller. SAC or TD3 can then be trained in a simulator containing randomized weather, demand, prices, and initial battery states. SAC is attractive because exploration and stochastic policies help with uncertainty; TD3 is a useful comparison.

Episodes may represent one day, but a time-limit truncation should not automatically be treated as a physical terminal state. The simulator should enforce hard state-of-charge and power limits. Training rewards should include degradation and constraint costs, while evaluation should report cost, peak demand, degradation proxy, violation rate, and robustness under unseen weather patterns.

Outcome and learning points

An acceptable deployment decision cannot be based only on average reward. A controller with slightly lower simulated cost but frequent constraint violations may be unacceptable. Stress tests should include forecast errors, sensor outages, extreme prices, and distribution shift. The case demonstrates that algorithm selection is only one part of RL engineering: formulation, simulator validity, safety, and evaluation determine whether an apparently successful policy is useful.

10. Ethical, Governance, and Deployment Issues

Deep RL systems can optimize measurable rewards while violating unmeasured human values. Reward hacking, unsafe exploration, discriminatory resource allocation, and opaque decisions are governance risks. Define who owns the reward specification, maintain versioned environments and policies, record training data and seeds, and document known failure modes.

Offline logs may be biased because they contain actions selected by an earlier policy. Evaluating a new policy from such data requires methods that account for missing counterfactual outcomes and distribution shift; naive average reward estimates can be misleading. Privacy and cybersecurity also matter when observations contain personal, industrial, or location data. Human override, audit logs, access controls, and rollback procedures should be designed before deployment.

11. Contemporary Directions and Best Practices

Current practice increasingly combines deep RL with offline datasets, demonstrations, world models, representation learning, recurrent memory, and constrained or risk-sensitive objectives. These methods can reduce costly exploration but introduce additional modeling assumptions. Generalization across tasks and environments remains a major challenge.

A practical checklist is:

• establish non-learning baselines;
• verify reward, termination, discounting, and action scaling;
• use target networks and replay correctly where appropriate;
• compare at least two suitable algorithm families;
• run multiple seeds and report uncertainty;
• evaluate out of distribution and under adversarial disturbances;
• monitor safety constraints independently of reward;
• retain reproducible configurations and checkpoints;
• deploy with a fallback policy and human oversight when consequences are significant.

Chapter Summary

Deep reinforcement learning uses neural networks to generalize RL beyond tabular state spaces. DQN combines Q-learning with replay buffers, target networks, and exploratory behavior; Double DQN reduces maximization bias, dueling networks separate state value from action advantage, prioritized replay focuses learning on significant transitions, and multi-step or distributional methods enrich targets. Policy gradients and actor–critic methods learn policies directly. PPO limits policy change through a clipped objective and is often stable but less sample efficient. DDPG applies deterministic actor–critic learning to continuous actions, while TD3 improves it with twin critics, delayed policy updates, and target smoothing. SAC adds entropy regularization and is a strong stochastic continuous-control method.

No algorithm guarantees stability. Performance depends on formulation, data distribution, reward scale, termination semantics, network and optimizer choices, and random seed. Sound engineering requires baselines, careful diagnostics, multiple-seed evaluation, safety constraints, and testing beyond the training distribution. In professional applications, deployment governance and failure management are as important as return optimization.

Key Terms

Deep reinforcement learning: RL using neural networks to approximate policies, values, or both.

DQN: A neural-network implementation of Q-learning for discrete actions.

Replay buffer: A memory of transitions sampled for training.

Target network: A slowly changing network used to construct bootstrap targets.

Temporal-difference error: The difference between a target return and a current value estimate.

Double DQN: A DQN variant that separates next-action selection from evaluation.

Dueling network: An architecture that estimates state value and action advantage separately.

Prioritized replay: Sampling transitions according to estimated learning significance.

Policy gradient: An optimization method based on gradients of log policy probabilities.

Actor–critic: A system containing a policy actor and a value-function critic.

Advantage function: The relative value of an action compared with the state value.

PPO: An on-policy algorithm using a clipped policy-ratio objective.

DDPG: A deterministic off-policy actor–critic algorithm for continuous actions.

TD3: A DDPG extension using twin critics, delayed updates, and target smoothing.

SAC: A maximum-entropy actor–critic algorithm for stochastic continuous control.

Bootstrapping: Estimating a target using another learned value estimate.

Sample efficiency: Performance gained per environment interaction.

Reward hacking: Achieving high specified reward through unintended behavior.

On-policy learning: Learning from data generated by the current policy.

Off-policy learning: Learning from data generated by a different behavior policy.

Review Questions

Short Answer Questions

1. Why are replay buffers and target networks useful in DQN?

2. What problem does Double DQN address?

3. Distinguish on-policy from off-policy learning.

4. What is the purpose of clipping in PPO?

5. Name the three principal improvements introduced by TD3 over DDPG.

Long Answer Questions

1. Explain the DQN learning cycle, including action selection, replay sampling, target construction, and target-network updates.

2. Compare DQN, PPO, TD3, and SAC in terms of action spaces, data usage, exploration, and stability.

3. Explain how bootstrapping, function approximation, and correlated data can destabilize deep RL.

4. Describe the role of the critic in actor–critic learning and analyze how critic error can affect the actor.

5. Design a reproducible evaluation protocol for a deep RL controller intended for deployment.

Analytical / Application Questions

1. A DQN transition has reward 1.5, γ=0.95, and next-state target values 2.0, 1.2, and 2.4. Calculate the nonterminal target and explain how it changes if the transition is terminal.

2. Select between PPO, TD3, and SAC for a continuous-control problem with expensive environment interactions and noisy dynamics. Justify the choice and identify two risks.

3. Design the state, action, reward, constraints, and evaluation metrics for an RL-based battery controller. Explain how you would test it against reward hacking and distribution shift.

Multiple Choice Questions with Answers

Question 1

Which mechanism primarily reduces correlation between consecutive training samples in DQN?

A. Experience replay

B. Action clipping

C. Entropy regularization

D. Reward discounting

Correct Answer: A. Experience replay

Question 2

In Double DQN, which network selects the next action?

A. The target network

B. The online network

C. A randomly initialized network

D. The replay buffer

Correct Answer: B. The online network

Question 3

What is the main purpose of a DQN target network?

A. To increase the number of actions

B. To provide a more slowly changing bootstrap target

C. To eliminate the need for exploration

D. To normalize rewards

Correct Answer: B. To provide a more slowly changing bootstrap target

Question 4

PPO clipping is designed primarily to:

A. Prevent policy ratios from encouraging excessively large updates

B. Make all rewards positive

C. Replace the critic with a table

D. Guarantee optimal exploration

Correct Answer: A. Prevent policy ratios from encouraging excessively large updates

Question 5

Which algorithm uses twin critics and delayed actor updates?

A. DQN

B. PPO

C. TD3

D. REINFORCE

Correct Answer: C. TD3

Question 6

For a nonterminal transition with r=1.5, γ=0.95, and maximum next-state value 2.4, what is the target?

A. 2.28

B. 3.78

C. 3.90

D. 1.55

Correct Answer: B. 3.78

Question 7

What additional objective is central to SAC?

A. Policy entropy

B. Episode length only

C. Action discreteness

D. Exact model reconstruction

Correct Answer: A. Policy entropy

Question 8

Which method is generally on-policy?

A. PPO

B. DQN

C. TD3

D. SAC

Correct Answer: A. PPO

Question 9

Why can standard Q-learning overestimate action values when estimates are noisy?

A. The maximum operator tends to select positively overestimated values

B. The discount factor is always zero

C. Replay removes all variance

D. The policy cannot select greedy actions

Correct Answer: A. The maximum operator tends to select positively overestimated values

Question 10

Which metric directly measures performance per environment interaction?

A. Sample efficiency

B. Network depth

C. Action-space cardinality

D. Target update frequency

Correct Answer: A. Sample efficiency