Statistical Learning Theory and Learning Paradigms

Introduction

Machine learning is often introduced as the design of algorithms that discover patterns from data. Statistical learning theory provides the deeper foundation for understanding when, why, and under what conditions such discovery can generalize beyond the observed examples. It connects optimization, probability, statistics, computation, and decision-making.

A central problem is the following. Let X denote the input space, Y the output space, and let (X, Y) be generated according to an unknown joint distribution P. A learning algorithm receives a finite sample S = {(x1, y1), ..., (xn, yn)} drawn from P and selects a predictive function h from a hypothesis space H. The goal is not merely to perform well on S, but to perform well on new observations drawn from the same population.

This chapter develops empirical risk minimization, structural risk minimization, hypothesis spaces, PAC learning, VC dimension, and inductive bias. It then compares major learning paradigms: supervised, unsupervised, semi-supervised, self-supervised, online, active, and transfer learning. The emphasis is on conceptual precision, practical selection of paradigms, and the limitations that arise when theoretical assumptions do not match real-world data.

Learning Objectives

After completing this chapter, a student should be able to:

1. Distinguish empirical risk from expected or population risk.

2. Explain empirical risk minimization and structural risk minimization.

3. Analyze the role of hypothesis spaces and inductive bias.

4. Interpret PAC learning and VC dimension at a postgraduate level.

5. Compare major machine-learning paradigms and identify suitable use cases.

6. Evaluate generalization, data requirements, computational constraints, and governance risks.

7. Apply these concepts to the design of a practical learning system.

1. The Statistical Learning Problem

Suppose a model receives an input X and produces a prediction h(X). A loss function L(y, h(x)) measures the cost of predicting h(x) when the true output is y. Examples include squared loss for regression and zero–one loss for classification.

The population risk, also called expected risk, is

R(h) = E[(X,Y)~P][L(Y, h(X))].

This is the quantity that matters operationally: the expected loss on future observations from P. It is generally unknown because P is unknown.

Given a training sample S of size n, the empirical risk is

R_hat_S(h) = (1/n) sum from i=1 to n of L(yi, h(xi)).

The empirical risk is observable and can be optimized. However, it may differ substantially from population risk, especially when n is small, the hypothesis space is highly flexible, or the observations are dependent.

The generalization gap is

R(h) - R_hat_S(h).

In practice, the true population risk is estimated using validation or test data. A test set should be used sparingly and kept separate from model development; repeated adaptation to test performance effectively turns the test set into training data.

Risk, noise, and the Bayes benchmark

For a given input x, uncertainty may arise because the relationship between X and Y is intrinsically noisy. The Bayes predictor minimizes population risk among all measurable prediction rules. Its risk is the Bayes risk, which represents the irreducible error under the selected loss and data-generating process.

A useful conceptual decomposition is:

prediction error = approximation error + estimation error + optimization error + irreducible noise.

Approximation error occurs when the selected hypothesis space cannot represent the underlying relationship. Estimation error arises from learning parameters from finite data. Optimization error results when the training procedure fails to find a suitable solution. Irreducible noise remains even for an ideal predictor.

This decomposition is conceptual rather than a universal algebraic identity; its exact form depends on the loss, model class, and assumptions.

2. Hypothesis Spaces and Inductive Bias

A hypothesis is a candidate predictive function. A hypothesis space H is the set of functions that a learning algorithm is permitted to consider. For example:

• H may consist of all linear functions h(x) = w^T x + b.
• H may consist of decision trees with maximum depth five.
• H may consist of neural networks with a specified architecture and parameter range.
• H may consist of probability models satisfying particular distributional assumptions.

The hypothesis space includes both the model family and, often, restrictions imposed through architecture, regularization, feature representation, or training procedures.

Inductive bias

Inductive bias is the set of assumptions that enables a learner to generalize from finite observations. No learner can infer a unique correct rule from finite data without assumptions. Two models that fit the training data equally well can make very different predictions outside the observed sample.

Examples of inductive bias include:

• Linearity: the relationship is approximately linear.
• Smoothness: nearby inputs tend to have similar outputs.
• Sparsity: only a small number of features are relevant.
• Locality: nearby pixels or tokens have related effects.
• Invariance: predictions should remain stable under transformations such as translation or rotation.
• Simplicity: among consistent explanations, simpler ones are preferred.

Inductive bias is not necessarily a weakness. It makes learning possible. The professional challenge is to ensure that the bias is appropriate for the application. A translation-invariant convolutional architecture is useful for image recognition but may be unsuitable for a task in which absolute position is essential.

Representation and hypothesis space

Feature engineering and representation learning alter the effective hypothesis space. A linear classifier applied to a well-designed nonlinear representation may express relationships that are impossible for a linear classifier on raw inputs. Consequently, model capacity cannot be assessed solely by counting visible parameters or by naming the algorithm; the representation, optimization procedure, and regularization all matter.

3. Empirical Risk Minimization

Empirical risk minimization (ERM) chooses a hypothesis with the smallest observed training loss:

h_hat_ERM in arg min over h in H of R_hat_S(h).

The principle is attractive because it is direct and often computationally tractable. Ordinary least squares is an ERM procedure using squared loss over a linear hypothesis space. Maximum likelihood estimation can also be viewed as ERM when the loss is negative log-likelihood.

Strengths of ERM

ERM is simple, general, and compatible with many loss functions. It provides a clear objective for optimization and is often statistically justified when the hypothesis space is controlled and the sample is representative.

Limitations of ERM

If H is excessively flexible, ERM may select a function that memorizes the training sample. Training loss can approach zero while population risk remains high. This is classical overfitting. Conversely, a severely restricted H can produce high training and test error, indicating underfitting.

ERM also depends on the quality of the loss function. A model trained with squared error may be poorly aligned with a business objective involving asymmetric financial costs. Data leakage, distribution shift, class imbalance, mislabeled examples, and dependence between observations can also invalidate naive ERM reasoning.

Regularized ERM

A common formulation adds a penalty:

h_hat in arg min over h in H of [R_hat_S(h) + lambda Omega(h)].

Here, Omega(h) measures model complexity or undesirable structure, and lambda is a nonnegative regularization coefficient. In linear regression, an L2 penalty discourages large coefficients, while an L1 penalty encourages sparsity. Regularization can be interpreted as restricting effective complexity, expressing a prior preference, or controlling optimization.

The penalty must be selected without using the final test set. Cross-validation or a validation set is commonly used, although these procedures require care when observations are temporal, clustered, or otherwise dependent.

4. Structural Risk Minimization

Structural risk minimization (SRM) addresses the tension between fit and complexity by organizing hypotheses into nested or otherwise ordered classes:

H1 subset of H2 subset of ... subset of HK.

A smaller class generally has lower capacity and greater approximation bias; a larger class generally offers greater expressive power but may have greater estimation error. SRM seeks a class and hypothesis that achieve a favorable balance between empirical performance and complexity.

Conceptually, one may select

k_hat = arg min over k of [R_hat_S(h_k) + complexity_penalty(H_k, n)],

where h_k is trained within H_k and the complexity penalty increases with the richness of H_k and decreases as sample size n increases. The exact penalty depends on the theoretical framework, such as VC bounds, covering numbers, Rademacher complexity, or information criteria.

SRM is broader than simply adding a parameter penalty. It can involve choosing tree depth, polynomial degree, network architecture, feature set, or a family of distributional assumptions. Modern deep learning complicates classical interpretations because highly overparameterized models can generalize well despite fitting the training data perfectly. This has motivated research into algorithmic stability, implicit regularization, margin-based explanations, data geometry, and the bias induced by optimization algorithms.

5. PAC Learning

Probably Approximately Correct (PAC) learning formalizes learnability probabilistically. Informally, a concept class is PAC learnable if, using a sufficient number of independent and identically distributed examples, a learner can produce a hypothesis whose error is at most epsilon with probability at least 1 minus delta.

Here:

• epsilon > 0 is the accuracy tolerance.
• delta > 0 is the failure probability.
• The guarantee is over the random draw of the training sample.
• The hypothesis is evaluated on the same underlying distribution assumed for training and testing.

For a binary classifier h, its true classification error is

err_P(h) = P(X,Y)[h(X) is not equal to Y].

A PAC guarantee has the form

P_S[err_P(h_hat_S) <= epsilon] >= 1 - delta.

The phrase “probably approximately correct” reflects the two parts of the guarantee: probably means confidence 1 − delta, and approximately correct means error at most epsilon.

Important assumptions

Classical PAC analyses commonly assume independent and identically distributed examples and a well-defined target concept or an appropriate agnostic-learning setting. In real applications, observations may be temporally dependent, labels may be noisy, and the deployment distribution may differ from the training distribution. Therefore, a PAC-style bound should not be interpreted as a guarantee under arbitrary distribution shift.

Realizable and agnostic settings

In the realizable setting, the target concept is assumed to belong to H, or there exists a zero-error hypothesis in H. In the agnostic setting, no such assumption is made; the learner seeks a hypothesis whose risk is close to the best achievable risk in H. The agnostic setting is more realistic for noisy data but generally requires more data or stronger complexity control.

6. VC Dimension and Capacity

The Vapnik–Chervonenkis (VC) dimension measures the expressive capacity of a binary hypothesis class. A set of points is shattered by H if, for every possible binary labeling of those points, there exists a hypothesis in H that produces that labeling. The VC dimension is the largest number of points that can be shattered, if such a finite maximum exists.

For example, affine linear classifiers in two-dimensional Euclidean space have VC dimension 3. They can realize every labeling of three points in general position, but cannot shatter every configuration of four points.

VC dimension helps explain why training accuracy alone is insufficient. A highly expressive class can represent many labelings and therefore can fit accidental patterns. Generalization bounds typically contain a complexity term involving VC dimension and a sample-size term. Their exact constants and forms vary, but the qualitative relationship is consistent: greater capacity requires more data for comparable generalization confidence.

A simplified conceptual bound for a binary class may be expressed as:

true error <= empirical error + complexity term + confidence term,

where the complexity term increases with VC dimension and decreases with n, and the confidence term increases as delta becomes smaller. This expression is intentionally schematic; precise bounds require specifying the theorem, loss, sample assumptions, and whether the setting is realizable or agnostic.

Limitations of VC dimension

VC dimension is powerful but not a complete practical measure of deep-model generalization. Modern neural networks may have enormous or even infinite effective capacity, yet generalize well under particular data distributions and optimization procedures. Other tools include Rademacher complexity, covering numbers, PAC-Bayes bounds, algorithmic stability, compression measures, margin analysis, and empirical scaling laws. These approaches are complementary rather than universally interchangeable.

7. Comparing Learning Paradigms

Learning paradigms differ mainly in the availability and structure of supervision, the timing of data arrival, and the relationship between source and target tasks.

7.1 Supervised learning

In supervised learning, the training set contains input–label pairs. Classification predicts discrete or categorical outcomes; regression predicts continuous quantities. Typical algorithms include linear models, support vector machines, decision trees, ensembles, and neural networks.

Supervised learning is appropriate when labels are available and the deployment objective is clearly defined. Its major cost is label acquisition. Labels may require experts, physical tests, manual review, or costly interventions. Label noise and inconsistent annotation can place a ceiling on performance.

Evaluation must reflect deployment conditions. For imbalanced classification, accuracy may be misleading; precision, recall, specificity, F1 score, calibration, expected cost, and area under an appropriate curve may be more informative. Metric selection should follow the consequences of errors.

7.2 Unsupervised learning

Unsupervised learning operates without externally supplied target labels. It seeks structure such as clusters, low-dimensional representations, density patterns, anomalies, or latent variables. Clustering, principal component analysis, matrix factorization, and density estimation are examples.

Unsupervised outputs are often exploratory rather than objectively correct. A cluster may be mathematically coherent but operationally meaningless. The number of clusters, distance metric, scaling, and representation can strongly affect results. Validation should therefore combine internal measures, stability analysis, domain knowledge, and downstream usefulness.

7.3 Semi-supervised learning

Semi-supervised learning uses a small labeled set together with a larger unlabeled set. It is useful when labels are expensive but raw observations are abundant. Common assumptions include smoothness, cluster structure, and that decision boundaries avoid dense regions.

Approaches include pseudo-labeling, consistency regularization, graph-based methods, and generative models. A risk is confirmation bias: incorrect early pseudo-labels may be reinforced. Unlabeled data can also be harmful when it comes from a different population or contains subgroups irrelevant to deployment.

7.4 Self-supervised learning

Self-supervised learning constructs supervisory signals from the data itself. In language modeling, a model may predict masked or subsequent tokens. In vision, it may predict relationships between transformed views or reconstruct masked regions. The learned representation is later adapted to one or more downstream tasks.

Self-supervision is especially valuable when raw data are plentiful and task labels are scarce. Its effectiveness depends on whether the pretext objective encourages representations useful for the target task. Low pretext loss does not automatically imply high downstream performance.

Self-supervised pretraining also introduces governance concerns. Large corpora may contain personal information, copyrighted material, toxic content, or systematic demographic bias. Data provenance, filtering, documentation, and evaluation are therefore essential.

7.5 Online learning

In online learning, data arrive sequentially and the model updates over time. It is suitable for fraud detection, recommendation, sensor monitoring, and changing markets. The learner may receive feedback after each prediction or after a delay.

Online learning must address concept drift, delayed labels, catastrophic adaptation to temporary anomalies, and computational limits. Evaluation should respect chronology: future observations must not influence past predictions. Prequential or time-ordered evaluation is often more appropriate than random cross-validation.

7.6 Active learning

Active learning allows the learner to select which examples should be labeled. The objective is to maximize performance per labeling cost. Query strategies include uncertainty sampling, diversity sampling, expected error reduction, and information-theoretic selection.

Active learning is useful when unlabeled examples are abundant and expert labeling is expensive. However, uncertainty sampling may overselect ambiguous or atypical cases, while ignoring important regions where the model is confidently wrong. Human reviewers may also be influenced by model suggestions, creating feedback loops.

7.7 Transfer learning

Transfer learning uses knowledge learned from a source task or domain to improve performance on a target task or domain. The source and target may differ in labels, input distributions, or both. Fine-tuning a pretrained language or vision model is a common example.

Transfer is beneficial when the source representation captures structure relevant to the target. Negative transfer occurs when source knowledge conflicts with target requirements. Evaluation must therefore compare against a target-only baseline and test performance on representative target data.

Domain adaptation is a related setting in which the task may remain similar while the input distribution changes. Domain generalization seeks performance on an unseen target domain without direct target training data. These problems are central in safety-critical and globally deployed systems.

8. A Practical Selection Framework

A learning paradigm should be selected by examining five questions:

1. What is the decision or prediction objective?

2. Which labels exist, and how reliable and costly are they?

3. How much unlabeled data are available, and do they represent deployment conditions?

4. Does data arrive in batches or streams, and is distribution shift expected?

5. What constraints govern privacy, fairness, latency, interpretability, and accountability?

A production system may combine paradigms. For example, self-supervised pretraining can be followed by supervised fine-tuning, active learning can prioritize expert labels, and online calibration can address changing prevalence. Such combinations should be evaluated as complete socio-technical systems rather than as isolated algorithms.

9. Illustrative Case: Predictive Maintenance in a Manufacturing Plant

Background

Consider an illustrative case involving a manufacturer operating hundreds of industrial pumps. Each pump generates vibration, temperature, pressure, and flow measurements. Unexpected failure causes production downtime, but expert technicians can label only a small number of historical failure events.

Problem

The organization wants to predict whether a pump is likely to fail within the next seven days. Failures are rare, sensor distributions vary between facilities, and the system must adapt as equipment ages.

Analysis

A purely supervised approach is constrained by the small number of labeled failures. Random train–test splitting would also be inappropriate if measurements from the same pump and future time periods appear in both sets. Such leakage could produce unrealistically high performance.

The project therefore uses time-ordered splits and groups observations by pump. Labels are defined using a fixed prediction horizon, and maintenance records are audited for inconsistency. The cost of a missed failure is substantially greater than the cost of an unnecessary inspection, so accuracy is not the principal metric.

Approach and solution

First, self-supervised representation learning is applied to large volumes of unlabeled sensor sequences. The pretraining task involves reconstructing masked segments and distinguishing temporally coherent windows from mismatched windows. The representation is then fine-tuned using the limited failure labels.

Second, active learning identifies cases for technician review. Selection combines predictive uncertainty with diversity across pump types and operating regimes. This avoids spending the entire labeling budget on near-duplicate ambiguous cases.

Third, a calibrated classifier produces risk estimates. A decision threshold is chosen using expected operational cost rather than a default probability of 0.5. Online monitoring tracks changes in sensor distributions, failure prevalence, calibration, and subgroup performance across facilities.

Finally, transfer learning is evaluated carefully. A model pretrained at one facility is compared with a model trained only on the target facility. If operating conditions differ substantially, unrestricted fine-tuning may cause negative transfer; facility-specific calibration or partial retraining may be preferable.

Outcome

The organization obtains a more data-efficient development process and a monitoring plan that recognizes distribution shift. The system is not accepted solely because of high retrospective discrimination. Deployment requires evidence that alerts arrive early enough to support maintenance, that technicians can understand the alert context, and that false alarms do not cause unsafe work practices.

Learning points

The case demonstrates that learning paradigms can be combined. It also shows why statistical learning theory must be connected to data collection, evaluation design, operational costs, and governance. A model can have low empirical risk while failing because the training sample is unrepresentative, labels are delayed, or the deployment distribution changes.

10. Challenges, Governance, and Best Practices

Distribution shift

The standard assumption that training and deployment data are identically distributed is often unrealistic. Covariate shift changes P(X), label shift changes P(Y), and concept shift changes P(Y|X). Detection may involve statistical monitoring, drift dashboards, recalibration tests, and domain expertise, but no single drift statistic proves that a model remains safe.

Fairness and subgroup performance

Average risk can conceal severe errors for minority or vulnerable groups. Evaluation should report relevant subgroup metrics, calibration behavior, data coverage, and uncertainty. Fairness objectives can conflict mathematically and operationally; the appropriate criterion depends on the context, legal framework, and consequences of errors.

Privacy and security

Training data may contain personal or commercially sensitive information. Access controls, minimization, retention limits, anonymization or pseudonymization, secure training procedures, and documented lawful purposes are important. Machine-learning systems are also exposed to adversarial examples, poisoning, membership inference, and model extraction.

Reproducibility and documentation

A credible learning system should document data provenance, inclusion criteria, labeling procedures, preprocessing, model versions, hyperparameter selection, evaluation splits, known limitations, and monitoring responsibilities. Model cards, data sheets, experiment tracking, and reproducible pipelines support technical and regulatory review.

Best-practice checklist

• Define the deployment decision before choosing a model.
• Align the loss and evaluation metrics with real consequences.
• Separate training, validation, and final test data.
• Use time-aware or group-aware splits when ordinary independence fails.
• Establish a simple baseline before adopting complex methods.
• Measure calibration as well as ranking or classification performance.
• Audit labels, missingness, subgroup coverage, and potential leakage.
• Treat unlabeled data as potentially biased, not automatically beneficial.
• Compare transfer and target-only models to detect negative transfer.
• Monitor drift, performance, fairness, and operational outcomes after deployment.

11. Contemporary Directions

Current research increasingly studies generalization beyond classical finite-dimensional settings. Topics include implicit bias in gradient-based optimization, scaling laws for foundation models, representation quality, distribution-free uncertainty, causal representation learning, federated and privacy-preserving learning, continual learning, and robust adaptation.

Foundation models demonstrate the practical power of self-supervised and transfer learning, but they also intensify concerns about compute concentration, environmental cost, data provenance, hallucination, evaluation leakage, and accountability. Statistical learning theory remains relevant because larger models do not eliminate the need to understand sampling assumptions, capacity, uncertainty, and deployment shift.

Chapter Summary

Statistical learning theory studies how a model learned from finite observations can generalize to future data. Population risk is the expected loss under the data-generating distribution, whereas empirical risk is the average loss on the observed sample. Empirical risk minimization selects the best-fitting hypothesis within a space, while structural risk minimization explicitly balances fit and hypothesis complexity.

Hypothesis spaces encode inductive bias, which is necessary for generalization but must be appropriate to the application. PAC learning expresses generalization probabilistically through accuracy and confidence requirements. VC dimension measures the shattering capacity of binary hypothesis classes and provides a foundation for reasoning about capacity, although modern deep learning requires complementary concepts such as algorithmic stability, margins, PAC-Bayes analysis, and implicit regularization.

Supervised learning uses labeled examples; unsupervised learning discovers structure without target labels; semi-supervised learning combines limited labels with unlabeled data; self-supervised learning creates supervisory signals from raw data; online learning adapts to streams; active learning selects examples for labeling; and transfer learning reuses knowledge across tasks or domains. These paradigms can be combined, but each introduces assumptions and risks.

Effective machine-learning design requires more than minimizing training loss. It requires representative data, appropriate evaluation, control of leakage, attention to distribution shift, fairness and privacy safeguards, operational monitoring, and clear accountability.

Key Terms

Hypothesis: A candidate predictive function considered by a learning algorithm.

Hypothesis space: The set of functions available to the learner.

Population risk: Expected loss on observations drawn from the underlying data distribution.

Empirical risk: Average loss measured on a finite training sample.

Generalization gap: Difference between population risk and empirical risk.

Empirical risk minimization: Selection of a hypothesis minimizing observed sample loss.

Structural risk minimization: Selection balancing empirical fit with hypothesis complexity.

Regularization: A method for discouraging undesirable or overly complex solutions.

Inductive bias: Assumptions that allow a learner to generalize beyond observed examples.

PAC learning: A framework requiring low error with high probability using sufficient data.

VC dimension: The largest number of points a binary hypothesis class can shatter.

Capacity: The expressive ability of a hypothesis class.

Approximation error: Error caused by limitations of the selected hypothesis space.

Estimation error: Error caused by learning from a finite sample.

Supervised learning: Learning from input–label pairs.

Unsupervised learning: Learning structure from data without target labels.

Semi-supervised learning: Learning from a combination of labeled and unlabeled data.

Self-supervised learning: Learning representations using targets generated from the data itself.

Online learning: Sequential learning from data arriving over time.

Transfer learning: Reusing knowledge from a source task or domain for a target task or domain.

Review Questions

Short Answer Questions

1. Distinguish population risk from empirical risk.

2. What is inductive bias, and why is it necessary for learning?

3. Define PAC learning using epsilon and delta.

4. What does VC dimension measure?

5. State one important difference between semi-supervised and self-supervised learning.

Long Answer Questions

1. Explain empirical risk minimization and structural risk minimization, including their advantages and limitations.

2. Discuss how hypothesis-space design, representation, and regularization affect generalization.

3. Explain the assumptions and limitations of classical PAC learning in real-world applications.

4. Compare supervised, unsupervised, semi-supervised, and self-supervised learning.

5. Compare online, active, and transfer learning in terms of data requirements, benefits, and risks.

Analytical / Application Questions

1. A fraud-detection dataset contains millions of transactions but very few confirmed fraud labels. Recommend a learning strategy and justify how labeling, evaluation, and drift monitoring should be designed.

2. A model performs very well under random cross-validation but poorly after deployment in another hospital. Analyze possible causes using distribution shift, inductive bias, and transfer-learning concepts.

3. Design a learning pipeline for predictive maintenance in a streaming industrial environment. Specify the learning paradigms, validation strategy, metrics, and governance controls you would use.

Multiple Choice Questions with Answers

Question 1

Which quantity directly represents expected performance on future observations drawn from the underlying distribution?

A. Empirical risk

B. Population risk

C. Training accuracy

D. Regularization penalty

Correct Answer: B. Population risk

Question 2

What is the main purpose of structural risk minimization?

A. To eliminate all training error

B. To maximize the number of model parameters

C. To balance empirical fit and hypothesis complexity

D. To avoid using a validation set

Correct Answer: C. To balance empirical fit and hypothesis complexity

Question 3

In PAC learning, epsilon represents the:

A. Failure probability

B. Accuracy or error tolerance

C. Number of training examples

D. VC dimension

Correct Answer: B. Accuracy or error tolerance

Question 4

A set of points is shattered when a hypothesis class can:

A. Classify every possible labeling of those points

B. Classify only linearly separable labels

C. Guarantee zero population risk

D. Use exactly one hypothesis

Correct Answer: A. Classify every possible labeling of those points

Question 5

Which statement best describes inductive bias?

A. Random noise added during training

B. Assumptions that support generalization from finite data

C. The final test-set score

D. The number of labels in a dataset

Correct Answer: B. Assumptions that support generalization from finite data

Question 6

Which paradigm creates supervisory targets from the data itself?

A. Self-supervised learning

B. Active learning

C. Online learning

D. Unsupervised clustering only

Correct Answer: A. Self-supervised learning

Question 7

Which learning paradigm allows the model to select examples for expert labeling?

A. Transfer learning

B. Active learning

C. Semi-supervised learning

D. Batch learning

Correct Answer: B. Active learning

Question 8

Why can random train–test splitting be inappropriate for time-dependent data?

A. It always increases model bias

B. It may allow future information to influence training evaluation

C. It prevents regularization

D. It removes all labels

Correct Answer: B. It may allow future information to influence training evaluation

Question 9

Negative transfer occurs when:

A. A source model improves target performance

B. A target dataset contains no features

C. Source knowledge harms performance on the target task

D. A model is trained online

Correct Answer: C. Source knowledge harms performance on the target task

Question 10

Which factor generally increases the risk of overfitting under empirical risk minimization?

A. A smaller and more restricted hypothesis space

B. A highly flexible hypothesis space with limited data

C. A larger representative test set

D. Lower measurement noise only

Correct Answer: B. A highly flexible hypothesis space with limited data