Unsupervised and Reinforcement Learning

Introduction and Context

Machine learning is commonly divided into supervised, unsupervised, and reinforcement learning according to the information available during learning. In supervised learning, examples are paired with target labels. In unsupervised learning, the system receives observations without target labels and seeks useful structure in them. In reinforcement learning (RL), an agent interacts with an environment, receives rewards or penalties, and learns behavior that improves long-term outcomes.

These paradigms address different practical problems. A retailer may want to discover customer segments without predefined categories; a manufacturer may need to identify unusual sensor behavior; and a robot may have to learn how to navigate while avoiding collisions. Unsupervised learning is primarily concerned with discovering structure in data, whereas reinforcement learning is concerned with sequential decision-making.

This chapter develops the principal methods used in both areas, explains their mathematical foundations, and examines their applications, limitations, and governance implications.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain the purpose and assumptions of unsupervised learning.

2. Apply and interpret clustering methods, including k-means and hierarchical clustering.

3. Explain dimensionality reduction using principal component analysis (PCA).

4. Identify anomalies and evaluate anomaly-detection systems.

5. Describe agents, states, actions, rewards, policies, and value functions in reinforcement learning.

6. Explain the Q-learning algorithm and the exploration–exploitation trade-off.

7. Select appropriate methods for practical AI applications and recognize their limitations and risks.

1. Unsupervised Learning: Discovering Structure Without Labels

Unsupervised learning uses a dataset \(X = \{x_1,x_2,\ldots,x_n\}\) in which each observation has features but no supplied target variable. The aim is not necessarily to find one objectively correct answer. Instead, the algorithm produces a representation, grouping, or score that helps a person or downstream system understand the data.

Typical goals include:

• Clustering: grouping observations that are similar under a chosen similarity measure.
• Dimensionality reduction: representing high-dimensional data with fewer variables while retaining important information.
• Anomaly detection: identifying observations that differ substantially from the expected pattern.
• Association discovery: finding variables or events that frequently occur together.

Unsupervised results depend strongly on feature representation, scaling, distance measures, and the purpose of analysis. A cluster found using income and age may differ from one found using purchasing frequency and product diversity. Thus, an unsupervised model does not discover meaning independently of the analyst's design choices.

Similarity, Distance, and Preprocessing

Many methods require a measure of similarity. For two vectors \(x=(x_1,\ldots,x_p)\) and \(y=(y_1,\ldots,y_p)\), Euclidean distance is

\[

d(x,y)=\sqrt{\sum_{j=1}^{p}(x_j-y_j)^2}.

\]

This measure is appropriate when numerical features are meaningfully comparable and squared differences are suitable. It is sensitive to scale: a variable measured in thousands can dominate a variable measured between 0 and 1. Standardization commonly transforms feature \(x_j\) into

\[

z_j=\frac{x_j-\bar{x}_j}{s_j},

\]

where \(\bar{x}_j\) is the sample mean and \(s_j\) is the sample standard deviation. Standardization is useful when variables have different units and comparable relative variation is intended. It may be inappropriate when the original scale has substantive meaning or when outliers make the mean and standard deviation unstable.

Categorical variables require appropriate encodings and distances. One-hot encoding can make a category difference count as a fixed numerical distance, while text and image data often require learned or domain-specific representations. Missing values should be handled before distance-based modeling because an undefined feature comparison can distort results.

2. Clustering Methods

2.1 k-Means Clustering

The k-means algorithm partitions observations into \(k\) clusters. It seeks centroids \(\mu_1,\ldots,\mu_k\) and assignments \(c(i)\) that minimize within-cluster sum of squared Euclidean distances:

\[

J=\sum_{i=1}^{n}\left\|x_i-\mu_{c(i)}\right\|^2.

\]

The objective assumes that numerical features, Euclidean distance, and compact, approximately spherical clusters are appropriate. The algorithm alternates between two steps:

1. Assign each observation to the nearest centroid.

2. Recalculate each centroid as the mean of observations assigned to it.

These steps continue until assignments or the objective change negligibly. Each iteration does not increase the objective, but the algorithm can converge to a local minimum. Therefore, multiple initializations are recommended. k-means++ improves initialization by selecting initial centroids that are spread across the data.

The value of \(k\) must be selected using subject knowledge, an elbow plot, silhouette analysis, or a business objective. The silhouette coefficient for an observation compares its average distance to its own cluster with its average distance to the nearest other cluster. A higher average silhouette score generally indicates better separation, although no internal metric proves that the clusters are meaningful.

#### Worked Example: One-Dimensional k-Means

Suppose standardized customer spending values are \(2, 3, 8, 9\), and \(k=2\). Start with centroids 2 and 8. The nearest-centroid assignments are:

• Cluster 1: 2 and 3
• Cluster 2: 8 and 9

The updated centroids are

\[

\mu_1=(2+3)/2=2.5,\qquad \mu_2=(8+9)/2=8.5.

\]

The assignments remain unchanged. The resulting objective is

\[

(2-2.5)^2+(3-2.5)^2+(8-8.5)^2+(9-8.5)^2=1.

\]

This is an excellent partition for these clearly separated values, but the result should not be generalized to data with irregular shapes, unequal densities, or substantial outliers.

2.2 Hierarchical Clustering

Hierarchical clustering produces a tree-like structure called a dendrogram. In agglomerative clustering, each observation begins as its own cluster. The closest pair is merged repeatedly until all observations form one cluster. A horizontal cut through the dendrogram determines the final number of clusters.

The definition of distance between clusters is called linkage:

• Single linkage: minimum pairwise distance; can create elongated chains.
• Complete linkage: maximum pairwise distance; tends to form compact groups.
• Average linkage: average pairwise distance.
• Ward linkage: merges clusters that cause the smallest increase in within-cluster squared error, usually with Euclidean data.

Hierarchical methods are useful when the analyst wants to inspect relationships at several granularities. They may be computationally expensive for very large datasets and are sensitive to the distance and linkage choices.

2.3 Density-Based Clustering

DBSCAN groups points in dense regions and labels points in sparse regions as noise. Its principal parameters are \(\varepsilon\), a neighborhood radius, and `minPts`, the minimum number of points required for a dense neighborhood. DBSCAN can identify non-spherical clusters and explicit noise, but its performance decreases when clusters have substantially different densities. Parameter selection is also sensitive to feature scaling.

Gaussian mixture models provide a probabilistic alternative. They assume observations arise from a mixture of probability distributions, commonly Gaussian components. Unlike k-means, a mixture model can provide soft membership probabilities. However, its assumptions about distributional shape may be unsuitable for complex data.

Evaluating Clusters

Cluster quality has two dimensions. Internal validation uses the data itself, through measures such as within-cluster dispersion or silhouette score. External validation compares clusters with an available reference, such as known categories, although such categories may defeat the original unsupervised purpose. Practical validation asks whether clusters are stable, interpretable, actionable, and useful for decisions.

A responsible analyst tests stability by resampling data, varying preprocessing choices, and trying reasonable algorithms. A visually attractive two-dimensional plot is not sufficient evidence of a reliable segmentation.

3. Dimensionality Reduction

High-dimensional datasets can contain redundant variables, noise, and correlations. Dimensionality reduction maps data from \(p\) variables to a smaller number \(r\) of components, with \(r<p\). It can support visualization, compression, denoising, and faster modeling.

Principal Component Analysis

PCA constructs orthogonal directions, called principal components, that successively maximize variance. Let \(X\) be a centered data matrix with \(n\) observations and \(p\) variables. Its sample covariance matrix is

\[

S=\frac{1}{n-1}X^TX.

\]

The first principal direction \(w_1\) solves

\[

\max_{\|w\|=1} w^T S w.

\]

The unit-length condition prevents the objective from being made arbitrarily large by scaling \(w\). The solution is the eigenvector of \(S\) associated with its largest eigenvalue \(\lambda_1\). Subsequent components are orthogonal to earlier components and correspond to descending eigenvalues. The variance explained by component \(j\) is

\[

\frac{\lambda_j}{\sum_{l=1}^{p}\lambda_l}.

\]

If variables have different units, PCA is often performed on standardized variables; otherwise, high-variance units may dominate. PCA captures linear patterns and is sensitive to outliers. Components are mathematical combinations, not automatically meaningful real-world variables.

#### Example of Explained Variance

Suppose the eigenvalues of a covariance matrix are 5, 3, 1, and 1. The total variance is 10. The first two components explain

\[

(5+3)/10=0.80,

\]

or 80% of the variance. Retaining two components may be reasonable for visualization or compression, but the unexplained 20% may still contain information important for a particular prediction task.

Other methods include t-SNE and UMAP, which are often useful for visualization of nonlinear relationships. Their plots should not be treated as faithful global maps: distances, cluster sizes, and apparent gaps can depend strongly on hyperparameters. An autoencoder can learn a lower-dimensional representation using neural networks, but interpretability and training stability require careful attention.

4. Anomaly Detection

An anomaly is an observation that deviates from a model of normal behavior. Anomalies may represent fraud, equipment failure, cyberattack, data-entry error, or a genuinely rare but important event. They are not automatically errors; rarity and undesirability are different concepts.

Methods include:

• Statistical methods: flag observations with unusually low probability under a fitted distribution. A univariate z-score, \((x-\bar{x})/s\), is useful only under appropriate distributional and scale assumptions.
• Distance-based methods: identify points far from a center or nearest neighbors.
• Density-based methods: compare local density with neighboring regions.
• Isolation Forest: recursively partitions data; unusual points tend to be isolated in fewer random splits.
• One-Class SVM: learns a boundary around a target population, subject to kernel and parameter choices.
• Autoencoders: flag observations with high reconstruction error, assuming the training data primarily represent normal behavior.

Evaluation is difficult because anomalies are usually rare and labels may be incomplete. Accuracy can be misleading: a system that labels every transaction as normal may achieve very high accuracy in a dataset with 0.1% fraud. Precision, recall, false-positive cost, detection delay, and analyst workload are more informative. Thresholds should reflect the relative cost of missed anomalies and unnecessary investigations.

5. Reinforcement Learning as Sequential Decision-Making

Reinforcement learning models an agent that repeatedly interacts with an environment. At time \(t\), the agent observes a state \(S_t\), selects an action \(A_t\), receives reward \(R_{t+1}\), and transitions to state \(S_{t+1}\). The objective is to maximize cumulative future reward rather than immediate reward alone.

Important terminology includes:

• State: information describing the current situation.
• Action: a choice available to the agent.
• Reward: scalar feedback indicating desirability of an outcome.
• Policy: a rule for choosing actions, written \(\pi(a|s)\).
• Return: cumulative discounted reward.
• Value function: expected return from a state or state–action pair.
• Environment: the external process producing transitions and rewards.
• Episode: a sequence ending in a terminal state, when applicable.

A common formal model is the Markov decision process (MDP), defined by states, actions, transition probabilities \(P(s'|s,a)\), reward expectations, and discount factor \(\gamma\), where \(0\leq\gamma<1\). The Markov property states that the next state depends on the current state and action, not the entire history. If observations do not contain enough information to satisfy this property, the problem may instead be a partially observable MDP.

For a policy \(\pi\), the return from time \(t\) is

\[

G_t=R_{t+1}+\gamma R_{t+2}+\gamma^2R_{t+3}+\cdots.

\]

The state-value function is

\[

V^\pi(s)=\mathbb{E}_\pi[G_t\mid S_t=s],

\]

and the action-value function is

\[

Q^\pi(s,a)=\mathbb{E}_\pi[G_t\mid S_t=s,A_t=a].

\]

The expectation is over environmental randomness and actions selected by the policy. A discount factor close to zero emphasizes immediate rewards; a value close to one gives greater weight to delayed rewards. In episodic tasks, undiscounted returns can sometimes be used when episodes are finite, but discounting is common for continuing tasks and mathematical convergence.

The optimal action-value function satisfies the Bellman optimality equation:

\[

Q^(s,a)=\mathbb{E}\left[R_{t+1}+\gamma\max_{a'}Q^(S_{t+1},a')\mid S_t=s,A_t=a\right].

\]

This equation expresses a central principle: the value of an action equals its immediate reward plus the discounted value of the best continuation. It assumes a well-defined MDP and a discount factor that supports finite values.

6. Q-Learning

Q-learning is an off-policy, model-free temporal-difference algorithm. It learns action values without explicitly estimating transition probabilities. After observing a transition \((s,a,r,s')\), it updates:

\[

Q(s,a)\leftarrow Q(s,a)+\alpha\left[r+\gamma\max_{a'}Q(s',a')-Q(s,a)\right],

\]

where \(\alpha\) is the learning rate, \(r\) is the observed reward, and the expression in brackets is the temporal-difference error. For a terminal next state, the future-value term is normally set to zero.

The target uses the maximum estimated value in the next state even if the behavior policy did not select that action; this is why Q-learning is off-policy. In contrast, SARSA updates toward the value of the action actually selected by the current policy and is on-policy.

#### Worked Q-Learning Update

Assume \(Q(s,a)=4\), observed reward \(r=2\), discount factor \(\gamma=0.9\), learning rate \(\alpha=0.5\), and the largest next-state value is 6. The target is

\[

2+0.9(6)=7.4.

\]

The temporal-difference error is \(7.4-4=3.4\). Therefore,

\[

Q_{new}(s,a)=4+0.5(3.4)=5.7.

\]

The update moves the estimate halfway toward the new target. It does not mean that 5.7 is the final value; repeated experience may continue to change it.

For finite state and action spaces, appropriate learning-rate conditions, and sufficient exploration, tabular Q-learning has strong convergence results under standard assumptions. In large or continuous spaces, a table is impractical, so function approximation—such as neural networks—is used. Deep Q-networks typically use experience replay and a target network to improve stability, but convergence is not guaranteed in the same straightforward way.

7. Exploration and Exploitation

An agent must exploit actions currently believed to be valuable while exploring actions whose value is uncertain. A simple epsilon-greedy policy chooses a random action with probability \(\varepsilon\) and a highest-valued action otherwise. If there are four actions and \(\varepsilon=0.2\), random exploration occurs 20% of the time; the exact probability assigned to the greedy action depends on how random actions are selected and whether ties exist.

Other approaches include softmax action selection, upper-confidence methods, and optimism in the face of uncertainty. Exploration schedules may reduce \(\varepsilon\) over time, but excessive reduction can prevent discovery of better strategies. In safety-critical applications, random exploration in the real environment may be unacceptable. Simulation, offline data, human oversight, constrained action spaces, and risk-sensitive objectives are safer alternatives.

Reward design is one of the most consequential engineering decisions. Sparse rewards can make learning slow; poorly chosen proxy rewards can produce reward hacking, in which the agent maximizes the numerical objective while violating the intended purpose. Reward functions should therefore be tested against adversarial and unusual behaviors, not only ordinary examples.

8. Applications and Professional Implications

Unsupervised learning is used for customer segmentation, document organization, image compression, patient profiling, network monitoring, and exploratory scientific analysis. Reinforcement learning is used in robotics, game playing, recommendation and advertising optimization, traffic signal control, inventory management, energy systems, and adaptive resource allocation.

In organizations, the key question is not simply whether an algorithm can be trained. Decision-makers must specify the objective, identify affected stakeholders, determine acceptable errors, and establish monitoring and accountability. Clusters may influence marketing access; anomaly scores may trigger account suspension; and RL policies may allocate scarce resources. Each use can create material consequences for people.

9. Illustrative Case: Predictive Maintenance in a Manufacturing Plant

Background

A manufacturing company operates industrial pumps equipped with vibration, temperature, pressure, and flow sensors. Unexpected pump failure causes production delays and expensive emergency repairs.

Problem

The company has only a small number of confirmed failure labels. It wants to discover operating regimes, detect unusual sensor patterns, and eventually recommend maintenance actions without allowing an automated system to endanger workers or equipment.

Analysis

Engineers first inspect sensor quality, synchronize timestamps, handle missing readings, and standardize variables. Clustering identifies several operating regimes, such as low-load, normal-load, and high-load operation. A single global anomaly threshold would be inappropriate because a high temperature may be normal during high-load operation but abnormal during low-load operation.

PCA is used to visualize correlated vibration and pressure measurements and to reduce noise for exploratory analysis. An Isolation Forest generates an anomaly score within each relevant operating context. Maintenance records are then reviewed to determine whether high-scoring events correspond to bearing wear, sensor faults, or harmless production changes.

Approach and Solution

The company deploys an anomaly-detection system as a decision-support tool rather than an autonomous shutdown mechanism. Alerts are ranked by severity and combined with operating context. A human maintenance engineer reviews alerts and records the outcome. These feedback records gradually create labels for a later supervised model.

RL is tested only in a digital simulation for scheduling inspections. The simulated agent receives rewards for reducing downtime and maintenance cost, with penalties for unsafe actions and excessive interventions. Before any operational trial, engineers validate the simulator against historical data and impose constraints such as maximum workload and mandatory safety inspections.

Outcome

The combined approach improves early investigation of unusual pump behavior and reduces unnecessary alerts compared with one fixed threshold. The organization also discovers that sensor drift, rather than mechanical failure, accounts for a substantial proportion of alerts. The RL scheduling policy remains advisory until simulation and controlled pilot evaluations demonstrate reliable performance.

Learning Points

1. Unsupervised methods are valuable when failure labels are scarce, but domain validation is essential.

2. Context-dependent normality is often more useful than one universal anomaly threshold.

3. Dimensionality reduction supports analysis but should not be mistaken for causal explanation.

4. RL should be constrained and tested in simulation when real-world exploration is risky.

5. Human review and feedback can improve both safety and future data quality.

10. Challenges, Ethics, and Governance

Technical Challenges

Unsupervised models can produce unstable results when data are noisy, sparse, nonstationary, or poorly represented. Choosing the number of clusters and interpreting components require judgment. Distribution shift can make a model trained on historical behavior unreliable after equipment, customers, or policies change.

RL faces delayed rewards, partial observability, unsafe exploration, nonstationary environments, and sample inefficiency. A policy may perform well in a simulator but fail in the real world because of the simulation-to-reality gap. Function approximation can amplify errors and create unpredictable behavior.

Fairness and Privacy

Clustering may create sensitive profiles or reproduce social segregation. Anomaly detection can disproportionately flag particular groups if historical data reflect biased surveillance or unequal access. Even when labels are absent, personal data can be identifiable. Data minimization, access controls, purpose limitation, and privacy-preserving techniques should be considered.

Accountability and Regulation

Organizations should document data sources, preprocessing, model versions, objectives, thresholds, and known limitations. High-impact systems require human oversight, appeal mechanisms, incident response, and monitoring for drift and disparate error rates. In regulated settings, applicable privacy, consumer-protection, employment, safety, and AI governance requirements must be incorporated into system design rather than added after deployment.

11. Best Practices

1. Define the operational purpose before selecting an algorithm.

2. Inspect, clean, scale, and document data transformations.

3. Compare multiple reasonable methods and test stability.

4. Use domain experts to interpret clusters and anomalies.

5. Evaluate rare-event systems with precision, recall, costs, and workload—not accuracy alone.

6. Keep training, validation, and operational monitoring processes distinct.

7. Use safe exploration, simulation, constraints, and human approval for RL applications.

8. Test for drift, unusual inputs, reward hacking, and adversarial behavior.

9. Record decisions, model versions, thresholds, and intervention outcomes.

10. Treat algorithmic output as evidence for a decision, not automatically as the decision itself.

Contemporary Developments

Current research increasingly combines self-supervised representation learning with clustering and anomaly detection, allowing systems to learn useful features from large unlabeled datasets. Deep RL is being combined with offline learning, imitation learning, world models, and multi-agent methods. Offline RL attempts to learn from previously collected interaction data, reducing risky online exploration, but it must handle distributional shift and actions poorly represented in the dataset.

Foundation models can provide representations for images, text, and sensor streams, after which unsupervised grouping or anomaly scoring is performed. Their flexibility does not eliminate the need for domain validation, privacy protection, and careful evaluation. Explainability research is also moving beyond visualizations toward uncertainty estimates, counterfactual analysis, and operationally meaningful explanations.

Chapter Summary

Unsupervised learning discovers structure without target labels. Clustering methods such as k-means, hierarchical clustering, DBSCAN, and mixture models differ in their assumptions about geometry, density, and membership. Data scaling, distance choice, initialization, and validation strongly influence results. PCA reduces dimensionality through orthogonal directions that maximize variance, although variance preservation is not equivalent to preserving task relevance. Anomaly detection identifies deviations from modeled normality and must be evaluated using error costs, rare-event metrics, and human impact.

Reinforcement learning addresses sequential decisions through interaction among an agent, environment, states, actions, and rewards. Policies define behavior; value functions estimate expected returns; and the Bellman equations express the recursive structure of decision value. Q-learning updates action values using observed rewards and estimated future value. Exploration must be balanced with exploitation, and unsafe real-world experimentation should be replaced or constrained by simulation, offline data, and human oversight.

Both paradigms are application-oriented but require careful problem definition, responsible data practices, monitoring, and governance. Useful AI systems depend not only on algorithmic performance but also on interpretability, safety, fairness, privacy, and accountable deployment.

Key Terms

Unsupervised learning: Learning from data without supplied target labels.

Clustering: Grouping observations according to a similarity or distance criterion.

Centroid: The mean position representing a cluster in k-means.

k-means: A clustering algorithm minimizing within-cluster squared Euclidean distance.

Hierarchical clustering: Clustering that builds a tree of nested groups.

DBSCAN: A density-based method that identifies dense groups and noise points.

Dimensionality reduction: Representing data with fewer variables or components.

PCA: A linear method that creates orthogonal directions ordered by explained variance.

Anomaly detection: Identifying observations that depart from expected behavior.

Markov decision process: A formal model of states, actions, transitions, rewards, and discounting.

Agent: The decision-making entity in reinforcement learning.

Reward: Numerical feedback from the environment.

Policy: A rule or probability distribution for selecting actions.

Return: Cumulative, often discounted, future reward.

Value function: Expected return associated with a state or action.

Q-function: Expected return for taking an action in a state and continuing thereafter.

Q-learning: Off-policy temporal-difference learning of optimal action values.

Exploration: Trying actions to obtain information about their outcomes.

Exploitation: Selecting actions currently believed to be valuable.

Reward hacking: Achieving a formal reward while violating the intended objective.

Review Questions

Short Answer Questions

1. Why can feature scaling affect the result of distance-based clustering?

2. What is the purpose of the two alternating steps in k-means?

3. What does the first principal component represent?

4. Distinguish a policy from a value function.

5. Why is Q-learning described as off-policy?

Long Answer Questions

1. Compare k-means, hierarchical clustering, and DBSCAN, including their assumptions and suitable applications.

2. Explain PCA mathematically and discuss why standardization may be necessary before applying it.

3. Describe anomaly detection methods and explain why accuracy is often a poor evaluation metric for rare events.

4. Explain the components of a Markov decision process and the meaning of discounted return.

5. Discuss the exploration–exploitation trade-off and the risks of poorly designed reward functions.

Analytical / Application Questions

1. A retailer has customer data containing annual spending, number of purchases, and average order value, all measured on different scales. Design a clustering workflow and justify your preprocessing, method selection, and validation strategy.

2. A fraud detector flags 2% of transactions, of which 20% are confirmed fraud. The system misses 10% of all confirmed fraud cases. Explain which performance measures should be examined and how the investigation threshold might be chosen.

3. A warehouse robot receives +10 for delivery, −1 for each time step, and −100 for collision. Analyze how this reward design may influence behavior and propose safeguards before using reinforcement learning in the real warehouse.

Multiple Choice Questions with Answers

Question 1

Which situation is most appropriate for unsupervised learning?

A. Predicting a labeled patient's diagnosis

B. Discovering customer groups without predefined segment labels

C. Estimating house prices from historical targets

D. Classifying emails using labeled spam examples

Correct Answer: B. Discovering customer groups without predefined segment labels

Question 2

What objective does standard k-means minimize?

A. Between-cluster classification error

B. Within-cluster sum of squared distances to centroids

C. Number of principal components

D. Total anomaly probability

Correct Answer: B. Within-cluster sum of squared distances to centroids

Question 3

Why is feature standardization often important before Euclidean-distance clustering?

A. It guarantees the correct number of clusters

B. It removes all outliers

C. It prevents variables with large units from dominating distance

D. It converts an unsupervised problem into a supervised problem

Correct Answer: C. It prevents variables with large units from dominating distance

Question 4

What does PCA order its components by?

A. Increasing number of observations

B. Decreasing explained variance

C. Increasing classification accuracy

D. Decreasing number of features

Correct Answer: B. Decreasing explained variance

Question 5

A dataset has PCA eigenvalues 6, 2, and 2. What proportion of variance is explained by the first component?

A. 20%

B. 40%

C. 60%

D. 80%

Correct Answer: C. 60%

Question 6

In reinforcement learning, what is a reward?

A. A rule for choosing actions

B. Numerical feedback supplied by the environment

C. A complete model of transition probabilities

D. A compressed state representation

Correct Answer: B. Numerical feedback supplied by the environment

Question 7

Which expression is the Q-learning update target for a nonterminal transition?

A. The current Q-value only

B. The reward minus the learning rate

C. The reward plus the discounted maximum next-state Q-value

D. The average of all past rewards without discounting

Correct Answer: C. The reward plus the discounted maximum next-state Q-value

Question 8

Why is Q-learning called off-policy?

A. It cannot use rewards

B. It learns only from unlabeled data

C. Its update evaluates the best next action even if the behavior policy did not select it

D. It never explores actions

Correct Answer: C. Its update evaluates the best next action even if the behavior policy did not select it

Question 9

What is the central purpose of exploration in reinforcement learning?

A. To avoid receiving any reward

B. To gather information about uncertain actions

C. To permanently select the current best action

D. To reduce the state space to one state

Correct Answer: B. To gather information about uncertain actions

Question 10

Which is a major risk of deploying reinforcement learning directly in a safety-critical physical environment?

A. The agent may explore unsafe actions

B. The agent cannot represent rewards numerically

C. The environment cannot contain states

D. Policies never change after initialization

Correct Answer: A. The agent may explore unsafe actions