Mathematical and Statistical Foundations

Introduction and Context

Advanced machine learning is not primarily a collection of algorithms; it is an application of mathematical models, statistical reasoning, and computational procedures to infer patterns from data. A sound foundation in these disciplines enables practitioners to understand why a model works, when its assumptions fail, how its parameters are estimated, and how its performance can be improved without merely increasing computational resources.

Machine-learning systems typically represent data as vectors or matrices, express relationships through functions, estimate unknown quantities from samples, and optimize an objective function. These activities require linear algebra, probability, statistics, calculus, information theory, and numerical optimization. They also require an understanding of generalization: the ability of a model to perform well on previously unseen observations.

This chapter develops the mathematical and statistical foundations needed for postgraduate study of machine learning. It emphasizes interpretation, assumptions, derivations, computational implications, and practical judgment.

Learning Objectives

By the end of this chapter, you should be able to:

1. Represent machine-learning data and transformations using vectors, matrices, and tensors.

2. Explain eigenvalues, eigenvectors, orthogonality, rank, and singular value decomposition in machine-learning contexts.

3. Use probability distributions, conditional probability, Bayes’ theorem, expectation, variance, and covariance to model uncertainty.

4. Distinguish estimation, prediction, sampling variability, bias, variance, and generalization error.

5. Explain entropy, cross-entropy, and Kullback–Leibler divergence.

6. Derive basic gradient-based learning rules and describe numerical optimization methods.

7. Analyze regularization, model complexity, computational complexity, and the bias–variance trade-off.

8. Apply these ideas to the design and evaluation of a machine-learning system.

1. Mathematical Representation of Data

1.1 Scalars, vectors, matrices, and tensors

A scalar is a single numerical value, such as a temperature or loss value. A vector is an ordered collection of values. For an observation with p features, the feature vector is commonly written as

x = [x₁, x₂, ..., xₚ]ᵀ ∈ Rᵖ.

A dataset containing n observations can be represented by a design matrix X ∈ Rⁿˣᵖ, where row i contains the features of observation i. If a model includes an intercept, a column of ones is often appended to X. A target vector is denoted y ∈ Rⁿ.

A matrix is a linear transformation between vector spaces. Matrix multiplication is defined only when dimensions are compatible. If A ∈ Rᵐˣⁿ and B ∈ Rⁿˣᵖ, then AB ∈ Rᵐˣᵖ. This dimensional discipline is important in implementation because many machine-learning errors are caused by incorrect shapes rather than incorrect statistical ideas.

Higher-dimensional arrays are called tensors. Images, for example, may be represented as height × width × channel tensors. The mathematical operations used in deep learning are generally extensions of matrix operations, although their computational implementations exploit specialized hardware.

1.2 Inner products, norms, and distance

The Euclidean inner product of two vectors is

xᵀz = Σᵢ xᵢzᵢ.

The induced Euclidean norm is

||x||₂ = √(xᵀx) = √(Σᵢ xᵢ²).

The squared Euclidean distance between x and z is ||x − z||₂². The L₁ norm is

||x||₁ = Σᵢ |xᵢ|,

and the infinity norm is ||x||∞ = maxᵢ |xᵢ|. Norms quantify magnitude and are also used to penalize model parameters. L₂ penalties generally shrink coefficients smoothly, whereas L₁ penalties can produce exact zeros and therefore encourage sparsity.

The cosine similarity between nonzero vectors is

cos(x,z) = (xᵀz)/(||x||₂||z||₂).

It measures orientation rather than magnitude and is frequently used in information retrieval and representation learning.

1.3 Linear transformations, rank, and conditioning

A transformation f(x) = Ax is linear if it preserves vector addition and scalar multiplication. The rank of A is the dimension of its column space. A matrix with deficient rank contains redundant linear information, which can make parameter estimation non-unique or numerically unstable.

The condition number of an invertible matrix, under the Euclidean norm, is

κ(A) = ||A||₂ ||A⁻¹||₂.

For a symmetric positive-definite matrix, this equals the ratio of its largest to smallest eigenvalue. A large condition number indicates sensitivity to perturbations and often causes slow convergence for gradient-based optimization. Feature scaling and regularization can improve conditioning.

1.4 Eigenvalues and eigenvectors

A nonzero vector v is an eigenvector of square matrix A if

Av = λv,

where λ is the corresponding eigenvalue. Eigenvectors identify directions whose orientation is preserved by A; eigenvalues specify the scaling in those directions.

For a covariance matrix, eigenvectors give principal directions of variation and eigenvalues give the associated variances. This interpretation underlies principal component analysis (PCA). Eigen-decomposition is most directly applicable to square matrices with appropriate structural properties, while singular value decomposition applies more generally.

1.5 Singular value decomposition and PCA

Every real m × n matrix X has a singular value decomposition (SVD)

X = UΣVᵀ,

where U and V have orthonormal columns or form orthogonal bases, and Σ contains nonnegative singular values σ₁ ≥ σ₂ ≥ ... ≥ 0.

For centered data matrix X, the sample covariance matrix is

S = (1/(n − 1))XᵀX.

The right singular vectors of X are eigenvectors of XᵀX. The first principal component is the unit vector v maximizing the projected sample variance:

maximize ||Xv||₂² subject to ||v||₂ = 1.

Since ||Xv||₂² = vᵀXᵀXv, the solution is the eigenvector associated with the largest eigenvalue of XᵀX. Subsequent components are constrained to be orthogonal to earlier components.

PCA is useful for visualization, compression, denoising, and reducing multicollinearity. It is unsupervised and maximizes variance, not necessarily predictive relevance. Features should usually be centered, and scaling is appropriate when variables have incomparable units or substantially different variances.

2. Probability and Uncertainty

2.1 Random variables and distributions

A random variable maps outcomes of an uncertain process to numerical values. A discrete random variable has a probability mass function p(x), while a continuous random variable has a density f(x). A density is not itself a probability at a point; probabilities are obtained by integration over intervals.

The expectation of a discrete variable is

E[X] = Σₓ x p(x),

and for a continuous variable,

E[X] = ∫ x f(x) dx,

provided the expectation exists. The variance is

Var(X) = E[(X − E[X])²] = E[X²] − E[X]².

The standard deviation is the square root of variance and has the same units as X.

For random variables X and Y, covariance is

Cov(X,Y) = E[(X − E[X])(Y − E[Y])].

Correlation is

Corr(X,Y) = Cov(X,Y)/(σₓσᵧ),

when both standard deviations are positive. Correlation measures linear association, not causation or general dependence.

2.2 Conditional probability and Bayes’ theorem

Conditional probability is defined, for P(B) > 0, as

P(A|B) = P(A ∩ B)/P(B).

Bayes’ theorem follows directly:

P(A|B) = P(B|A)P(A)/P(B).

In classification, A may denote a class and B an observed feature vector. P(A) is the prior probability, P(B|A) is the class-conditional likelihood, and P(A|B) is the posterior probability. The denominator can be expanded using the law of total probability:

P(B) = Σₖ P(B|Cₖ)P(Cₖ),

for mutually exclusive and exhaustive classes Cₖ.

A naive Bayes classifier assumes conditional independence of features given the class:

P(x|Cₖ) = Πⱼ P(xⱼ|Cₖ).

This assumption is often false, but the resulting classifier can still perform well, especially in high-dimensional text applications.

2.3 Common distributions

The Bernoulli distribution models one binary trial with success probability p. Its mean is p and variance is p(1 − p). A binomial random variable counts successes in n independent Bernoulli trials and has probability

P(X=k) = C(n,k)pᵏ(1−p)ⁿ⁻ᵏ.

The categorical distribution models one outcome among multiple classes. The multinomial distribution counts outcomes across repeated categorical trials.

The Gaussian distribution with mean μ and variance σ² has density

f(x) = [1/(σ√(2π))] exp(−(x−μ)²/(2σ²)),

where σ > 0. Gaussian assumptions are convenient but should not be adopted solely because they simplify analysis. Heavy-tailed, skewed, or multimodal data may require other distributions or robust methods.

2.4 Law of large numbers and central limit theorem

The law of large numbers states, informally, that under suitable conditions the sample mean converges to the population mean as sample size increases. The central limit theorem states that, under broad conditions, a standardized sample mean approaches a Gaussian distribution as sample size grows, even when the original observations are not Gaussian.

These results support confidence intervals, approximate hypothesis tests, and many statistical learning arguments. They do not eliminate dependence, sampling bias, distribution shift, or data leakage.

3. Statistical Learning and Estimation

3.1 Population, sample, and empirical risk

Let (X,Y) be a random input–output pair and let fθ be a model with parameters θ. Given loss function ℓ, the population risk is

R(θ) = E[ℓ(fθ(X),Y)].

Because the data-generating distribution is unknown, training minimizes empirical risk over a sample D of size n:

R̂(θ) = (1/n)Σᵢ₌₁ⁿ ℓ(fθ(xᵢ),yᵢ).

Empirical risk minimization is effective only when the sample is informative about the population and the model class is appropriately controlled. A model can achieve very low training loss while having high population risk.

3.2 Maximum likelihood and least squares

Suppose observations are conditionally independent given parameters θ, with likelihood p(yᵢ|xᵢ;θ). The likelihood is

L(θ) = Πᵢ p(yᵢ|xᵢ;θ).

Maximum likelihood estimation chooses θ maximizing L(θ), or equivalently maximizing the log-likelihood

log L(θ) = Σᵢ log p(yᵢ|xᵢ;θ).

For linear regression with yᵢ = xᵢᵀβ + εᵢ and independent Gaussian errors εᵢ ~ N(0,σ²), maximizing likelihood with respect to β is equivalent to minimizing the residual sum of squares:

Σᵢ (yᵢ − xᵢᵀβ)² = ||y − Xβ||₂².

If XᵀX is invertible, the ordinary least-squares solution is

β̂ = (XᵀX)⁻¹Xᵀy.

This formula is primarily theoretical in modern software; QR or SVD methods are numerically more stable than explicitly computing an inverse. The formula also fails to provide a unique ordinary least-squares solution when XᵀX is singular.

3.3 Sampling bias and leakage

Statistical validity depends on how data are collected. Selection bias occurs when the sample systematically differs from the target population. Measurement bias arises from flawed instruments or labeling procedures. Data leakage occurs when information unavailable at prediction time enters training or validation, producing an unrealistically optimistic estimate.

Train–validation–test separation, time-aware splitting for temporal data, and preprocessing fitted only on training data are essential safeguards.

4. Calculus and Optimization

4.1 Derivatives, gradients, and Hessians

For a scalar function f of one variable, the derivative measures local rate of change. For f: Rᵖ → R, the gradient is

∇f(θ) = [∂f/∂θ₁, ..., ∂f/∂θₚ]ᵀ.

The gradient points in the direction of steepest local increase under the Euclidean norm. The Hessian is the matrix of second partial derivatives:

H(θ) = ∇²f(θ).

A twice-differentiable function is locally convex at a point if its Hessian is positive semidefinite there. A function is convex over a region if its Hessian is positive semidefinite throughout that region, when the Hessian exists.

4.2 Gradient descent

For differentiable objective J(θ), gradient descent updates parameters as

θₜ₊₁ = θₜ − ηₜ∇J(θₜ),

where ηₜ > 0 is the learning rate. The method assumes that the gradient provides a useful local direction and that the learning rate is sufficiently controlled. Too large a rate can cause divergence; too small a rate can cause slow progress.

For a quadratic objective J(θ) = 1/2 θᵀAθ − bᵀθ, where A is symmetric positive definite, the minimizer satisfies Aθ = b. Gradient descent converges for a constant learning rate satisfying

0 < η < 2/λmax(A),

where λmax(A) is the largest eigenvalue. This illustrates the effect of conditioning: if eigenvalues vary greatly, progress may be rapid in some directions and slow in others.

4.3 Stochastic and mini-batch optimization

If J(θ) is an average over n examples, stochastic gradient methods estimate the full gradient using a randomly selected example or mini-batch. Under suitable sampling and step-size conditions, these estimates can be unbiased or approximately unbiased. Mini-batches reduce computational cost per update and exploit parallel hardware, but introduce gradient noise.

Momentum methods accumulate a moving direction, while adaptive methods scale updates using historical gradient magnitudes. These methods can improve practical convergence but may have different generalization behavior and require careful hyperparameter selection.

4.4 Newton and quasi-Newton methods

Newton’s method uses curvature:

θₜ₊₁ = θₜ − [H(θₜ)]⁻¹∇J(θₜ),

when the Hessian is invertible. Near a well-behaved optimum, it can converge rapidly. However, computing and storing the Hessian is expensive for high-dimensional models, and an indefinite Hessian can produce a non-descent direction. Quasi-Newton methods approximate curvature using gradient history and are often useful for medium-scale problems.

5. Information Theory and Machine Learning

5.1 Entropy

For a discrete random variable X with probability mass function p(x), Shannon entropy is

H(X) = −Σₓ p(x)log p(x).

The logarithm base determines units: base 2 gives bits, while the natural logarithm gives nats. Entropy measures uncertainty in a distribution. It is largest for the uniform distribution over a fixed finite support.

For a classification target with class probabilities p₁,...,p_K, entropy is

H(Y) = −Σₖ pₖ log pₖ.

A pure node has entropy zero; a node with evenly balanced classes has higher entropy.

5.2 Cross-entropy and log loss

Given true distribution p and predicted distribution q over the same classes, cross-entropy is

H(p,q) = −Σₖ pₖ log qₖ.

For a one-hot target whose correct class is c, this reduces to −log q_c. Assigning a very small probability to the correct class produces a large penalty. This makes cross-entropy sensitive to both classification accuracy and probability calibration.

For a model producing qθ(y|x), minimizing average negative log-likelihood is equivalent to minimizing cross-entropy between empirical targets and predictions. This is why cross-entropy is standard for probabilistic classification.

5.3 Kullback–Leibler divergence

The KL divergence from p to q is

D_KL(p||q) = Σₓ p(x)log[p(x)/q(x)],

provided q(x) > 0 wherever p(x) > 0. It is nonnegative and equals zero if and only if p and q are equal almost everywhere, but it is not symmetric and therefore is not a distance metric. The identity

H(p,q) = H(p) + D_KL(p||q)

shows that, when p is fixed, minimizing cross-entropy is equivalent to minimizing KL divergence.

6. Bias, Variance, Generalization, and Regularization

6.1 Bias–variance decomposition

For squared-error prediction at a fixed input x, let the observed response be Y = f(x) + ε, where E[ε|x] = 0 and Var(ε|x) = σ². If a learning algorithm trained on a random dataset produces prediction f̂_D(x), then expected test mean squared error can be decomposed as

E_D,Y[(Y − f̂_D(x))²] = [E_D f̂_D(x) − f(x)]² + E_D[(f̂_D(x) − E_D f̂_D(x))²] + σ².

The three terms are squared bias, variance, and irreducible noise. The decomposition depends on squared loss and the stated noise assumptions. It should not be applied unchanged to classification accuracy or arbitrary losses.

A highly restricted model may have high bias and low variance. A highly flexible model may have low bias and high variance. Increasing training data often reduces variance, while improving representation or model structure may reduce bias.

6.2 Regularization

Regularization modifies the training objective to discourage undesirable parameter values. Ridge regression solves

minβ ||y − Xβ||₂² + λ||β||₂²,

where λ ≥ 0. For λ > 0, the solution is

β̂ridge = (XᵀX + λI)⁻¹Xᵀy,

when the inverse exists. The intercept is commonly excluded from the penalty. Ridge regression reduces coefficient variance and handles collinearity, at the cost of introducing shrinkage bias.

Lasso regression solves

minβ ||y − Xβ||₂² + λ||β||₁.

The non-differentiability of the L₁ norm at zero contributes to sparse solutions. Regularization strength must be selected using validation procedures, and features should generally be standardized before comparing coefficient penalties.

6.3 Generalization and capacity

Generalization is influenced by model capacity, sample size, noise, optimization, data quality, and the relationship between training and deployment distributions. Classical learning theory uses concepts such as VC dimension, uniform convergence, and algorithmic stability to formalize some aspects of capacity and generalization.

Modern overparameterized models complicate the simple picture that test error must increase after interpolation. In some settings, test error follows a double-descent pattern: it decreases, rises near the interpolation threshold, and decreases again as model size increases. This does not invalidate regularization or validation; it emphasizes that optimization, implicit regularization, data structure, and architecture all influence generalization.

7. Computational Complexity and Numerical Reliability

Computational complexity describes how resource requirements scale with input size. Forming XᵀX for an n × p matrix typically requires O(np²) arithmetic operations, while solving a dense p × p linear system by standard factorization is approximately O(p³). Explicit matrix inversion should generally be avoided because solving systems directly is more stable and often more efficient.

Memory complexity is equally important. A dense p × p matrix requires O(p²) storage, which becomes prohibitive for large p. Sparse representations, iterative solvers, randomized SVD, and distributed computation can reduce resource demands.

Numerical issues include overflow, underflow, cancellation, ill-conditioning, and loss of precision. The log-sum-exp identity improves numerical stability when computing log probabilities:

log Σᵢ exp(aᵢ) = m + log Σᵢ exp(aᵢ−m), where m = maxᵢ aᵢ.

Subtracting m prevents excessively large exponentials without changing the mathematical result. Stable software should also use robust stopping criteria, monitor objective values and gradients, and test sensitivity to scaling and initialization.

8. Detailed Applied Case: Predicting Loan Default

The following is an illustrative case, not a report of a particular institution.

Background and problem

A financial institution wants to estimate the probability that an applicant will default within twelve months. Available variables include income, debt-to-income ratio, employment duration, credit history, and recent payment behavior. The output will support—not automatically determine—credit review.

Analysis

The target is binary, so a probabilistic classifier is appropriate. The institution must first define the prediction point and ensure that every feature would be available at that point. A payment status recorded after approval would be leakage. Historical data may also reflect previous discriminatory policies, creating representational and measurement concerns.

Continuous features have different scales. Standardization based only on the training set is appropriate for models sensitive to feature scale. Correlated variables may produce unstable coefficients in unregularized logistic regression, motivating ridge regularization or feature selection.

Approach and solution

The dataset is split chronologically when deployment is future-oriented. A training set is used to fit preprocessing and model parameters; a validation set selects the regularization strength; a final test set is used once for an unbiased performance estimate.

For logistic regression, the estimated probability is

P(Y=1|x) = 1/[1 + exp(−(β₀ + xᵀβ))].

The model is trained by minimizing regularized binary cross-entropy. Performance should be assessed using discrimination measures such as area under the ROC curve, but also calibration, precision–recall behavior, expected financial cost, and subgroup performance. A threshold should be chosen according to documented costs of false approvals and false rejections rather than defaulting to 0.5.

Outcome

Suppose validation shows that a regularized model has slightly lower accuracy than an unrestricted model but substantially better calibration and more stable subgroup performance. The regularized model is preferable if these properties better support responsible decisions. Deployment should include monitoring for drift in income distributions, repayment behavior, and approval outcomes.

Learning points

1. Statistical performance is inseparable from data collection and target definition.

2. Probability calibration can be more operationally useful than accuracy alone.

3. Regularization can improve stability, but it does not remove social or measurement bias.

4. Fairness assessment requires defined groups, metrics, contexts, and governance procedures.

5. Human oversight, documentation, appeal processes, and periodic review are essential in high-impact applications.

9. Limitations, Governance, and Contemporary Directions

Mathematical elegance does not guarantee valid deployment. Assumptions of independent observations, stable distributions, accurate labels, and representative samples may fail. A model may be well calibrated overall but poorly calibrated for a subgroup. Statistical parity metrics can conflict, especially when base rates differ, so fairness requires normative and institutional judgment rather than a single universal formula.

Privacy is also relevant. Data minimization, access control, secure computation, and appropriate anonymization reduce risk, although anonymization does not guarantee re-identification resistance. High-impact automated decisions may be subject to sector-specific regulation, documentation requirements, transparency obligations, and human review.

Contemporary machine learning increasingly uses automatic differentiation, accelerator hardware, probabilistic programming, causal modeling, uncertainty quantification, and foundation models. These developments do not replace foundational mathematics. Automatic differentiation still computes derivatives; probabilistic systems still depend on distributions and conditional assumptions; large models still face optimization, generalization, calibration, and computational-cost challenges.

Best practice is to make assumptions explicit, scale and validate data transformations correctly, use numerically stable implementations, compare against simple baselines, tune hyperparameters using principled validation, report uncertainty where possible, and monitor models after deployment.

Chapter Summary

Machine learning depends on a connected mathematical framework. Linear algebra represents observations, parameters, transformations, and low-dimensional structure. Probability models uncertainty and provides the language of conditional inference. Statistics distinguishes population objectives from sample-based estimates and clarifies sampling variability, estimation, and leakage. Calculus supplies gradients and curvature for optimization, while numerical methods determine whether theoretical procedures are computationally reliable. Information theory explains entropy, cross-entropy, and KL divergence, which are central to classification and probabilistic modeling. The bias–variance framework explains competing sources of prediction error, and regularization controls instability by trading variance for bias. Complexity analysis connects model design to time and memory constraints. Finally, responsible application requires attention to calibration, fairness, privacy, governance, and distribution shift.

Key Terms

Vector: An ordered collection of numerical values.

Design matrix: A matrix whose rows represent observations and columns represent features.

Norm: A function measuring the magnitude of a vector.

Rank: The dimension of a matrix’s column or row space.

Condition number: A measure of sensitivity of a numerical problem to perturbations.

Eigenvector: A nonzero vector whose direction is preserved by a linear transformation.

Singular value decomposition: Factorization of a matrix into orthogonal matrices and singular values.

Covariance: A measure of joint linear variation between two random variables.

Likelihood: The probability model viewed as a function of unknown parameters.

Empirical risk: Average loss computed on a finite dataset.

Gradient: Vector of first-order partial derivatives.

Hessian: Matrix of second-order partial derivatives.

Entropy: A measure of uncertainty in a probability distribution.

Cross-entropy: Expected negative log probability assigned by one distribution to outcomes from another.

KL divergence: A nonnegative, generally asymmetric measure of discrepancy between distributions.

Bias: Systematic error caused by restrictive assumptions or model structure.

Variance: Variation in a fitted model across different training samples.

Regularization: Penalization or constraint used to control model complexity.

Generalization: Performance on data not used for model fitting.

Data leakage: Use of information during training that would not be available at prediction time.

Review Questions

Short Answer Questions

1. What is the role of the design matrix in supervised learning?

2. Distinguish covariance from correlation.

3. Why is explicit matrix inversion usually discouraged in numerical implementations?

4. What does the learning rate control in gradient descent?

5. Why can cross-entropy strongly penalize an incorrect high-confidence prediction?

Long Answer Questions

1. Explain SVD and derive its relationship to PCA for a centered data matrix.

2. Derive the bias–variance decomposition for squared-error prediction and state its assumptions.

3. Compare ridge and lasso regularization, including their effects on coefficients and practical use cases.

4. Explain maximum likelihood estimation and show why Gaussian linear regression leads to least squares.

5. Discuss how conditioning, scaling, and numerical stability affect machine-learning optimization.

Analytical / Application Questions

1. A model has low training error but high validation error. Analyze at least four possible causes and propose a diagnostic plan.

2. Design a validation strategy for a loan-default model trained on five years of data, identifying potential leakage, temporal dependence, and fairness concerns.

3. For a classification problem with predicted probabilities, explain how you would evaluate discrimination, calibration, decision thresholds, and subgroup reliability.

Multiple Choice Questions with Answers

Question 1

For a centered data matrix X, the first principal component is the unit vector that maximizes which quantity?

A. The determinant of X

B. The projected variance ||Xv||₂²

C. The L₁ norm of v without constraints

D. The number of observations

Correct Answer: B. The projected variance ||Xv||₂²

Question 2

Which condition is required for the ordinary least-squares expression (XᵀX)⁻¹Xᵀy to be uniquely defined?

A. X must have orthogonal rows

B. y must have zero mean

C. XᵀX must be invertible

D. All features must be binary

Correct Answer: C. XᵀX must be invertible

Question 3

What does the gradient of a differentiable scalar objective indicate locally?

A. The direction of steepest increase under the Euclidean norm

B. The exact global minimum

C. The model’s classification accuracy

D. The sample variance of the target

Correct Answer: A. The direction of steepest increase under the Euclidean norm

Question 4

Which term in the squared-error bias–variance decomposition represents irreducible observation noise?

A. Squared bias

B. Model variance

C. Regularization error

D. Noise variance

Correct Answer: D. Noise variance

Question 5

What is a characteristic effect of an L₁ penalty?

A. It always increases every coefficient

B. It can produce exactly zero coefficients

C. It eliminates the need for validation

D. It guarantees unbiased estimates

Correct Answer: B. It can produce exactly zero coefficients

Question 6

If p is a true distribution and q is a predicted distribution, cross-entropy is defined as:

A. Σₖ pₖ log qₖ

B. Σₖ qₖ log pₖ

C. −Σₖ pₖ log qₖ

D. Σₖ(pₖ−qₖ)² only

Correct Answer: C. −Σₖ pₖ log qₖ

Question 7

For a symmetric positive-definite matrix A, which quantity determines the upper bound on a constant gradient-descent step size for a quadratic objective?

A. The smallest eigenvalue only

B. The determinant only

C. The trace only

D. The largest eigenvalue

Correct Answer: D. The largest eigenvalue

Question 8

Which practice prevents preprocessing leakage during model evaluation?

A. Fit scaling parameters using the complete dataset before splitting

B. Fit preprocessing parameters only on the training data

C. Use the test set to select hyperparameters

D. Duplicate validation examples in training

Correct Answer: B. Fit preprocessing parameters only on the training data

Question 9

What does KL divergence measure?

A. The discrepancy between two probability distributions

B. The Euclidean distance between feature vectors

C. The number of model parameters

D. The rank of a covariance matrix

Correct Answer: A. The discrepancy between two probability distributions

Question 10

For an n × p dense design matrix, forming XᵀX typically has arithmetic complexity of approximately:

A. O(n+p)

B. O(p)

C. O(np²)

D. O(2ⁿ)

Correct Answer: C. O(np²)