Mathematical and Computational Foundations

Introduction and Context

Machine learning systems are built from mathematical objects and computational procedures. A dataset is represented numerically; a model transforms numerical inputs into predictions; a loss function measures error; and an optimization algorithm adjusts parameters to reduce that error. TensorFlow and PyTorch automate much of the calculation, but reliable use of these frameworks requires understanding what the computations mean.

This chapter develops the foundations needed to read, implement, debug, and evaluate machine-learning models. It connects linear algebra with data representation, probability and statistics with uncertainty and inference, calculus with learning, and numerical computation with practical software engineering.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish scalars, vectors, matrices, and higher-order tensors.

2. Perform and interpret common vector and matrix operations.

3. Explain probability, random variables, expectation, variance, covariance, and conditional probability.

4. Use descriptive statistics and understand sampling and data leakage.

5. Explain derivatives, partial derivatives, gradients, and the chain rule.

6. Formulate and interpret common loss functions.

7. Apply gradient descent and identify important optimization issues.

8. Explain computational graphs and automatic differentiation.

9. Recognize numerical stability, shape, datatype, and conditioning problems.

10. Connect these foundations to TensorFlow and PyTorch implementations.

1. Mathematical Objects for Machine Learning

1.1 Scalars, vectors, matrices, and tensors

A scalar is a single number, such as a learning rate or temperature. A vector is an ordered collection of numbers. A feature vector for one observation may be written as

\[

\mathbf{x} = [x_1,x_2,\ldots,x_d]^T \in \mathbb{R}^d.

\]

Here, \(d\) is the number of features. A matrix is a rectangular array. If there are \(n\) observations and \(d\) features, a data matrix may be represented as \(X\in\mathbb{R}^{n\times d}\), where row \(i\) contains observation \(i\).

A tensor generalizes these objects to any number of dimensions. Its number of dimensions is called its rank or order in many machine-learning contexts, although this usage differs from matrix rank. For example:

• scalar: shape \(()\);
• vector: shape \((d)\);
• matrix: shape \((n,d)\);
• colour image batch: shape \((b,h,w,c)\) in channels-last format, or \((b,c,h,w)\) in channels-first format.

Shape is not merely a programming detail. It expresses the semantic organization of the data and determines whether operations are valid.

1.2 Vector operations

For vectors of equal dimension, addition is componentwise:

\[

\mathbf{a}+\mathbf{b}=[a_1+b_1,\ldots,a_d+b_d].

\]

The dot product is

\[

\mathbf{a}^T\mathbf{b}=\sum_{j=1}^{d}a_jb_j.

\]

It produces a scalar and measures alignment. If two nonzero vectors have angle \(\theta\), then

\[

\mathbf{a}^T\mathbf{b}=\|\mathbf{a}\|_2\|\mathbf{b}\|_2\cos\theta,

\]

where the Euclidean norm is \(\|\mathbf{a}\|_2=\sqrt{\sum_j a_j^2}\). Cosine similarity is the dot product divided by the product of the norms, provided neither vector is zero.

A linear model uses an affine function

\[

z=\mathbf{w}^T\mathbf{x}+b,

\]

where \(\mathbf{w}\) is a parameter vector and \(b\) is a bias. Strictly, the bias makes the function affine rather than linear, but machine-learning literature often calls the complete model a linear model.

1.3 Matrix multiplication and transformations

If \(A\in\mathbb{R}^{m\times n}\) and \(B\in\mathbb{R}^{n\times p}\), their product \(AB\) exists and has shape \(m\times p\):

\[

(AB)_{ij}=\sum_{k=1}^{n}A_{ik}B_{kj}.

\]

Matrix multiplication is generally not commutative: \(AB\) need not equal \(BA\), and one product may be defined while the other is not. A matrix can represent a rotation, scaling, projection, or a layer that maps one feature space into another.

The transpose \(A^T\) exchanges rows and columns. A square matrix is invertible if there exists \(A^{-1}\) such that \(AA^{-1}=I\). In machine learning, explicit matrix inversion is usually avoided because it can be expensive and numerically unstable; solving a linear system is preferred.

1.4 Eigenvalues, rank, and positive semidefiniteness

An eigenvector \(\mathbf{v}\neq 0\) of a square matrix satisfies

\[

A\mathbf{v}=\lambda\mathbf{v},

\]

where \(\lambda\) is its eigenvalue. Eigenvalues help describe directions of stretching and are important in covariance analysis, principal component analysis, and optimization.

The rank of a matrix is the dimension of its column space. Low-rank structure can indicate redundancy and can motivate compression or dimensionality reduction.

A symmetric matrix \(A\) is positive semidefinite if

\[

\mathbf{z}^TA\mathbf{z}\geq 0

\]

for every vector \(\mathbf{z}\). Covariance matrices are positive semidefinite because they represent variances of linear combinations.

1.5 Broadcasting and tensor operations

Frameworks often support broadcasting, in which dimensions of size one are expanded conceptually to permit elementwise operations. For example, adding a vector of shape \((d)\) to every row of a matrix of shape \((n,d)\) is valid. Broadcasting is useful but can conceal errors. A tensor should therefore be checked for intended shape before and after an operation.

2. Probability and Statistics

2.1 Probability as a model of uncertainty

A probability model assigns nonnegative values to events, with total probability one. For a discrete random variable \(X\), a probability mass function \(p(x)=P(X=x)\) satisfies

\[

\sum_x p(x)=1.

\]

For a continuous variable, a density \(f(x)\) satisfies \(f(x)\geq0\) and \(\int_{-\infty}^{\infty}f(x)dx=1\). A density value is not itself a probability; probabilities are obtained by integrating over intervals.

The conditional probability of event \(A\) given event \(B\), when \(P(B)>0\), is

\[

P(A\mid B)=\frac{P(A\cap B)}{P(B)}.

\]

Bayes' theorem follows from this definition:

\[

P(A\mid B)=\frac{P(B\mid A)P(A)}{P(B)}.

\]

In classification, this distinguishes the prior probability of a class from the posterior probability after observing features.

2.2 Random variables and expectation

A random variable maps outcomes to numerical values. Its expectation is the probability-weighted average:

\[

\mathbb{E}[X]=\sum_x xp(x)

\]

for a discrete variable, or \(\mathbb{E}[X]=\int xf(x)dx\) for a continuous variable when the integral exists.

Variance measures spread around the mean \(\mu=\mathbb{E}[X]\):

\[

\operatorname{Var}(X)=\mathbb{E}[(X-\mu)^2]=\mathbb{E}[X^2]-\mu^2.

\]

The standard deviation is the square root of variance and has the same units as the variable. Covariance between \(X\) and \(Y\) is

\[

\operatorname{Cov}(X,Y)=\mathbb{E}[(X-\mathbb{E}[X])(Y-\mathbb{E}[Y])].

\]

Positive covariance indicates that the variables tend to increase together; zero covariance does not generally imply independence.

2.3 Common distributions

The Bernoulli distribution models one binary trial with success probability \(p\). A categorical distribution models one outcome from more than two categories. A binomial distribution models the number of successes in a fixed number of independent Bernoulli trials. The Gaussian distribution is frequently used as an approximation and as a noise model:

\[

f(x)=\frac{1}{\sigma\sqrt{2\pi}}\exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right),

\]

where \(\sigma>0\). The formula describes a density, not a probability at one exact continuous value.

2.4 Descriptive statistics and data splitting

The sample mean of values \(x_1,\ldots,x_n\) is

\[

\bar{x}=\frac{1}{n}\sum_{i=1}^{n}x_i.

\]

The commonly used unbiased sample variance is

\[

s^2=\frac{1}{n-1}\sum_{i=1}^{n}(x_i-\bar{x})^2,

\]

assuming the observations are an independent sample from a population and the goal is unbiased estimation of population variance.

A dataset is commonly divided into training, validation, and test sets. The training set estimates parameters; the validation set supports model selection and hyperparameter decisions; the test set provides a final, ideally unused estimate of generalization. Data leakage occurs when information unavailable at prediction time influences training or evaluation. For example, fitting a standardization mean using the entire dataset before splitting leaks information from validation or test data.

2.5 Standardization and normalization

A standardized feature is often computed as

\[

z_i=\frac{x_i-\mu_{\text{train}}}{\sigma_{\text{train}}},

\]

where statistics are computed on training data only and \(\sigma_{\text{train}}>0\). Standardization can improve optimization when features have very different scales. It does not guarantee a Gaussian distribution.

3. Calculus for Learning

3.1 Derivatives and partial derivatives

For a scalar function \(f(x)\), the derivative at \(x\) is the limiting rate of change

\[

f'(x)=\lim_{h\to0}\frac{f(x+h)-f(x)}{h},

\]

when the limit exists. For a function of several variables, a partial derivative varies one variable while holding the others constant.

The gradient of a scalar function \(f(\mathbf{w})\) is the column vector of partial derivatives:

\[

\nabla_{\mathbf{w}}f=\left[\frac{\partial f}{\partial w_1},\ldots,\frac{\partial f}{\partial w_d}\right]^T.

\]

The gradient points in the direction of greatest local increase. Therefore, negative gradient points toward greatest local decrease for sufficiently small steps.

3.2 Chain rule

If \(y=f(u)\) and \(u=g(x)\), then

\[

\frac{dy}{dx}=\frac{dy}{du}\frac{du}{dx}.

\]

For vector-valued intermediate variables, the analogous relationship uses Jacobian matrices. Neural networks are compositions of functions, so repeated application of the chain rule produces backpropagation.

3.3 Worked derivative example

Consider a single-observation squared-error model

\[

\hat{y}=wx+b,\qquad L=\frac{1}{2}(\hat{y}-y)^2.

\]

The factor \(1/2\) is conventional and simplifies differentiation. Let \(e=\hat{y}-y\). Since \(\partial L/\partial \hat{y}=e\), \(\partial\hat{y}/\partial w=x\), and \(\partial\hat{y}/\partial b=1\), the chain rule gives

\[

\frac{\partial L}{\partial w}=ex,\qquad \frac{\partial L}{\partial b}=e.

\]

For \(x=2\), \(y=5\), \(w=1\), and \(b=0\), the prediction is \(2\), the error is \(-3\), and the gradients are \(-6\) and \(-3\). A gradient-descent step with learning rate \(0.1\) gives \(w'=1.6\) and \(b'=0.3\). The update moves parameters in the direction opposite to the negative gradients.

4. Loss Functions and Statistical Meaning

A loss function assigns a numerical penalty to a prediction. Training usually minimizes an empirical risk, such as

\[

J(\theta)=\frac{1}{n}\sum_{i=1}^{n}\ell(f_\theta(x_i),y_i),

\]

where \(\theta\) denotes trainable parameters and \(\ell\) is the per-example loss.

4.1 Regression losses

Mean squared error is

\[

\operatorname{MSE}=\frac{1}{n}\sum_{i=1}^{n}(\hat{y}_i-y_i)^2.

\]

It strongly penalizes large errors and, under a conditional Gaussian noise model with constant variance, corresponds to maximum-likelihood estimation up to constants.

Mean absolute error is

\[

\operatorname{MAE}=\frac{1}{n}\sum_{i=1}^{n}|\hat{y}_i-y_i|.

\]

It is less sensitive to outliers than MSE, though it is not differentiable at zero; software can use subgradients or related smooth alternatives.

4.2 Classification losses

For binary classification with predicted probability \(p\in(0,1)\) and label \(y\in\{0,1\}\), binary cross-entropy is

\[

\ell=-[y\log p+(1-y)\log(1-p)].

\]

For multiclass classification with one-hot target \(\mathbf{y}\) and predicted probabilities \(\mathbf{p}\), categorical cross-entropy is

\[

\ell=-\sum_{k=1}^{K}y_k\log p_k.

\]

In practice, logits are often passed directly to a numerically stable combined softmax-cross-entropy operation rather than applying softmax and logarithm separately.

Loss is not identical to every evaluation metric. A model trained with cross-entropy may be evaluated using accuracy, recall, precision, calibration, or a cost-sensitive metric, depending on the application.

5. Optimization

5.1 Gradient descent

For parameters \(\theta\) and differentiable objective \(J\), basic gradient descent is

\[

\theta_{t+1}=\theta_t-\eta\nabla_\theta J(\theta_t),

\]

where \(\eta>0\) is the learning rate. The formula assumes the gradient is available and that a sufficiently small step is useful locally. It does not guarantee a global minimum, especially for nonconvex neural-network objectives.

Batch gradient descent uses all training examples. Stochastic gradient descent uses one example, while mini-batch gradient descent uses a subset. Mini-batches provide a practical balance between noisy estimates and hardware efficiency.

5.2 Convexity and local behavior

A differentiable function is convex when the line segment between any two points lies above the function graph. For convex optimization, every local minimum is global, although practical issues such as ill-conditioning may remain. Neural-network losses are generally nonconvex, with possible saddle points and flat or steep regions.

The Hessian matrix contains second derivatives. It describes local curvature and helps explain why one learning rate may be too large in a steep direction and too small in a flat direction.

5.3 Practical optimizers

Momentum maintains a moving direction to reduce oscillation. Adaptive methods such as Adam maintain estimates of first and second moments of gradients. They can converge effectively, but their hyperparameters, regularization behavior, and generalization properties differ from plain SGD. No optimizer removes the need for suitable data, scaling, architecture, validation, and monitoring.

Regularization modifies the objective or training process to discourage undesirable solutions. L2 regularization adds a term proportional to \(\|\theta\|_2^2\); dropout randomly removes activations during training; early stopping halts training when validation performance stops improving. These methods address overfitting but do not replace sound experimental design.

6. Computational Graphs and Automatic Differentiation

A computational graph represents a calculation as nodes and directed edges. Nodes may represent variables or operations. For example, the expression

\[

z=wx+b,\quad e=z-y,\quad L=\frac{1}{2}e^2

\]

can be represented as a sequence of operations. During the forward pass, values are computed. During the backward pass, derivatives are propagated using the chain rule.

Automatic differentiation is not the same as symbolic differentiation or finite differences. It decomposes a program into elementary operations and applies exact chain-rule calculations up to floating-point arithmetic. Reverse-mode automatic differentiation is especially efficient when a function has many inputs and one scalar output, as in neural-network loss computation.

In PyTorch, tensors can track operations when `requires_grad=True`; calling `loss.backward()` computes gradients, and an optimizer updates parameters. A typical conceptual pattern is:

```python

optimizer.zero_grad()

prediction = model(x)

loss = loss_function(prediction, target)

loss.backward()

optimizer.step()

```

In TensorFlow, `tf.GradientTape()` records operations and computes derivatives:

```python

with tf.GradientTape() as tape:

prediction = model(x)

loss = loss_function(target, prediction)

grads = tape.gradient(loss, model.trainable_variables)

optimizer.apply_gradients(zip(grads, model.trainable_variables))

```

The exact APIs evolve, but the mathematical workflow remains the same: forward computation, loss evaluation, differentiation, and parameter update.

7. Numerical Computation and Implementation Discipline

Computers use finite-precision representations. Floating-point arithmetic is approximate, and operations may produce rounding error, overflow, underflow, or cancellation. A mathematically valid expression can therefore fail numerically.

7.1 Stability practices

Use stable library implementations for log-sum-exp, softmax-cross-entropy, and probability calculations. For a set of logits \(a_1,\ldots,a_K\), the stable identity

\[

\log\sum_k e^{a_k}=m+\log\sum_k e^{a_k-m},\qquad m=\max_k a_k,

\]

avoids unnecessarily large exponentials.

Avoid computing \(\log(0)\); probabilities may need clipping when a separate probability-based calculation is unavoidable. Monitor for NaN and infinity values. Gradient clipping can limit excessively large gradients, but it should be investigated rather than used to conceal a defective model or data pipeline.

7.2 Shape, dtype, and device

Common implementation failures include mismatched batch dimensions, integer tensors used where floating-point gradients are required, and tensors placed on different devices. Labels for a classification loss may require a specific shape and integer type, whereas model outputs generally use floating-point values. Explicit assertions and small test batches are valuable safeguards.

A matrix operation should be checked dimensionally. If \(X\) has shape \((b,d)\) and \(W\) has shape \((d,h)\), then \(XW\) has shape \((b,h)\). Writing down shapes before coding often prevents silent broadcasting errors.

7.3 Reproducibility and performance

Set random seeds where appropriate, record package versions, preserve data-splitting procedures, and log hyperparameters. Reproducibility is probabilistic rather than absolute when parallel hardware and nondeterministic kernels are involved.

Vectorized tensor operations are generally faster than Python-level loops because they exploit optimized kernels and accelerators. However, performance should be measured. Premature optimization can make code harder to verify.

8. Detailed Applied Case: Predicting House Prices

The following is an illustrative case showing how the foundations combine in a practical workflow.

Background and problem

A property platform wants to estimate house prices from floor area, number of rooms, age, and distance to a transport station. The objective is a useful prediction for initial valuation, not an authoritative legal appraisal.

Analysis

The data are represented by a matrix \(X\in\mathbb{R}^{n\times4}\), with target vector \(\mathbf{y}\in\mathbb{R}^n\). Because floor area and distance may have different scales, training-set standardization is considered. The dataset is split into training, validation, and test partitions before fitting preprocessing statistics.

A baseline affine model is

\[

\hat{y}_i=\mathbf{w}^T\mathbf{x}_i+b.

\]

The training objective is MSE, possibly with an L2 penalty:

\[

J(\mathbf{w},b)=\frac{1}{n}\sum_{i=1}^{n}(\hat{y}_i-y_i)^2+\lambda\|\mathbf{w}\|_2^2,

\]

where \(\lambda\geq0\) controls the strength of the penalty. The penalty is applied deliberately and documented; it is not included in the evaluation metric unless that is explicitly intended.

Approach and solution

The team first inspects missing values, outliers, units, and possible leakage. For example, a feature derived from the final sale price would be invalid at prediction time. It standardizes numerical features using training statistics and trains a model with mini-batch gradient descent. Validation MSE and MAE are monitored, while the test set remains untouched until the model and preprocessing choices are finalized.

A residual analysis checks whether errors vary systematically by neighbourhood, price range, or property age. If the model underestimates expensive properties, transformations, additional features, or a nonlinear model may be considered. Any added complexity must be assessed against validation performance and operational requirements.

Outcome

Suppose the model reduces validation error relative to a mean-price baseline but performs poorly on a neighbourhood absent from training data. The result indicates that average predictive performance is not sufficient: the model may have limited geographic generalization. The deployment decision should therefore include a confidence or abstention policy, monitoring for distribution shift, and periodic reassessment.

Learning points

1. Mathematical representation determines what the model can learn.

2. Preprocessing must be fitted without using test information.

3. Loss selection reflects assumptions about error costs and noise.

4. Optimization quality depends on scaling, learning rate, and numerical stability.

5. A single aggregate metric can hide subgroup failures.

6. Responsible deployment requires monitoring, documentation, and human oversight.

9. Limitations, Governance, and Contemporary Practice

Mathematical correctness does not guarantee a valid or fair AI system. Training data may encode historical discrimination, measurement error, or unequal coverage. A model can be well optimized and still produce harmful outcomes if its objective is poorly chosen.

Professional practice requires documenting data provenance, intended use, known limitations, preprocessing, model versions, and evaluation populations. Sensitive applications may require privacy protection, access control, auditability, explainability, and compliance with applicable data-protection and sectoral regulations. Mathematical uncertainty should not be presented as certainty to decision-makers.

Contemporary systems increasingly use mixed precision, automatic compilation, distributed training, and hardware accelerators. These technologies improve speed and memory efficiency but introduce additional concerns: loss scaling, reproducibility, communication overhead, and device-specific numerical behavior. Understanding the underlying operations remains essential even when high-level libraries generate much of the computation.

Best-Practice Checklist

• Define the meaning and units of every feature.
• Record tensor shapes at model boundaries.
• Split data before learning preprocessing statistics.
• Select losses according to prediction goals and assumptions.
• Use stable library operations for exponentials and logarithms.
• Start with a simple baseline before adding complexity.
• Inspect gradients, losses, predictions, and validation behavior.
• Distinguish optimization performance from generalization performance.
• Test edge cases, missing values, and distribution shifts.
• Document experiments and evaluate impacts on relevant groups.

Chapter Summary

Vectors and matrices provide the language for feature representation and model transformations, while tensors organize batched, sequential, and spatial data. Probability describes uncertainty; expectation, variance, covariance, and conditional probability support statistical reasoning. Calculus supplies derivatives, gradients, and the chain rule needed to train parameterized models. Loss functions convert prediction quality into optimization objectives, and gradient-based methods update parameters to reduce those objectives. Computational graphs and automatic differentiation implement the chain rule efficiently in TensorFlow and PyTorch. Finally, numerical stability, correct shapes and datatypes, reproducibility, data governance, and subgroup evaluation are necessary for dependable machine-learning practice.

Key Terms

Scalar: A single numerical value.

Vector: An ordered one-dimensional collection of values.

Matrix: A rectangular two-dimensional array.

Tensor: A multidimensional numerical array.

Dot product: The sum of pairwise products of two equal-length vectors.

Norm: A function measuring the size of a vector.

Eigenvalue: A scalar describing how a matrix scales an associated eigenvector.

Random variable: A numerical mapping from outcomes of a random process.

Expectation: The probability-weighted average of a random variable.

Variance: The expected squared deviation from the mean.

Covariance: A measure of joint variation between two variables.

Gradient: The vector of partial derivatives of a scalar function.

Chain rule: A calculus rule for differentiating compositions of functions.

Loss function: A function measuring the penalty for a prediction.

Gradient descent: An iterative method that moves opposite the objective gradient.

Regularization: A method that discourages undesirable model complexity or parameters.

Computational graph: A representation of operations and dependencies in a computation.

Automatic differentiation: Chain-rule-based derivative computation through elementary operations.

Data leakage: Use of information in training that would not be available at prediction time.

Broadcasting: Implicit expansion of compatible tensor dimensions for an operation.

Review Questions

Short Answer Questions

1. What is the shape of the product of an \((m\times n)\) matrix and an \((n\times p)\) matrix?

2. Why must standardization statistics generally be computed from training data only?

3. Define the gradient of a scalar-valued function.

4. State one difference between MSE and MAE.

5. What is the purpose of a computational graph?

Long Answer Questions

1. Explain how vectors, matrices, and tensors represent data in a neural-network pipeline, including the importance of shape.

2. Explain expectation, variance, and covariance and discuss their relevance to machine learning.

3. Derive the gradients of the single-example squared-error model \(L=\frac12(wx+b-y)^2\) with respect to \(w\) and \(b\).

4. Compare batch, stochastic, and mini-batch gradient descent, including practical advantages and limitations.

5. Discuss numerical stability problems in exponential and logarithmic computations and describe appropriate remedies.

Analytical / Application Questions

1. A dataset contains features measured in metres, years, and currency units. Design a preprocessing and evaluation procedure that avoids leakage and supports reliable gradient-based training.

2. A classifier has high accuracy but poor recall for a minority class. Analyze why loss selection, class distribution, and evaluation metrics might produce this result, and propose an evaluation plan.

3. A neural network's training loss becomes NaN after several iterations. Develop a systematic debugging strategy using concepts from gradients, floating-point arithmetic, tensor shapes, datatypes, and learning-rate selection.

Multiple Choice Questions with Answers

Question 1

If X has shape (32, 10) and W has shape (10, 4), what is the shape of XW?

A. (32, 4)

B. (10, 32)

C. (4, 10)

D. (32, 10)

Correct Answer: A. (32, 4)

Question 2

Which statement about covariance is correct?

A. Zero covariance always implies independence.

B. Covariance has no relationship to variable scales.

C. Covariance measures joint variation between two variables.

D. Covariance must always be positive.

Correct Answer: C. Covariance measures joint variation between two variables.

Question 3

For L = 1/2(wx+b-y)^2, what is the derivative with respect to b?

A. x(wx+b-y)

B. wx+b-y

C. 1/2(wx+b-y)^2

D. y-wx-b

Correct Answer: B. wx+b-y

Question 4

Which operation is most directly associated with avoiding overflow in softmax-related calculations?

A. Subtracting the maximum logit before exponentiation

B. Multiplying all logits by zero

C. Converting labels to integers

D. Removing the bias term

Correct Answer: A. Subtracting the maximum logit before exponentiation

Question 5

What is the main role of a validation set?

A. To fit the model parameters exclusively

B. To replace the training set

C. To support model and hyperparameter selection

D. To guarantee unbiased predictions in deployment

Correct Answer: C. To support model and hyperparameter selection

Question 6

For a binary label y=1 and predicted probability p=0.8, binary cross-entropy is -log(0.8). Which interpretation is correct?

A. It is zero because the prediction is correct.

B. It is a positive penalty that decreases as p approaches 1.

C. It equals the model's accuracy.

D. It is defined only when y is continuous.

Correct Answer: B. It is a positive penalty that decreases as p approaches 1.

Question 7

Which update represents basic gradient descent?

A. theta_next = theta + learning_rate * gradient

B. theta_next = gradient - learning_rate * theta

C. theta_next = theta / gradient

D. theta_next = theta - learning_rate * gradient

Correct Answer: D. theta_next = theta - learning_rate * gradient

Question 8

Why is reverse-mode automatic differentiation suitable for a scalar neural-network loss?

A. It avoids all floating-point operations.

B. It efficiently computes derivatives with respect to many inputs or parameters.

C. It requires no forward computation.

D. It only works for linear models.

Correct Answer: B. It efficiently computes derivatives with respect to many inputs or parameters.

Question 9

Which practice prevents a common form of data leakage?

A. Fitting feature means and standard deviations on training data only

B. Using test labels during hyperparameter tuning

C. Combining training and test data before preprocessing

D. Selecting the best model using final test performance repeatedly

Correct Answer: A. Fitting feature means and standard deviations on training data only

Question 10

If an error is e = -3 and x = 2 in the single-example squared-error model, what is the gradient with respect to w?

A. -6

B. -1.5

C. 5

D. 6

Correct Answer: A. -6