5. Multivariable Calculus and Automatic Differentiation

Introduction and Context

Machine-learning models are functions that transform inputs into predictions. Training such a model requires more than evaluating the function: we must determine how its output changes when its parameters change. Calculus provides the language and computational procedures for making these determinations. In a scalar problem, the derivative measures the rate of change of one quantity with respect to another. Modern machine-learning systems, however, usually involve thousands, millions, or billions of interacting variables. Their analysis therefore depends on multivariable calculus.

This chapter develops partial derivatives, gradients, Jacobians, Hessians, Taylor approximations, multivariable chain rules, and constrained differentiation. These ideas culminate in backpropagation and automatic differentiation, the mechanisms that make gradient-based learning practical. The emphasis is not only on symbolic definitions but also on dimensions, interpretation, computational graphs, numerical stability, and professional use.

Learning Objectives

By the end of this chapter, you should be able to:

• compute and interpret partial derivatives, directional derivatives, gradients, Jacobians, and Hessians;
• use vector and matrix notation consistently in differentiation;
• apply first- and second-order Taylor approximations;
• derive composite functions using the multivariable chain rule;
• explain constrained differentiation using Lagrange multipliers;
• distinguish numerical, symbolic, and automatic differentiation;
• derive the backpropagation rule for layered models;
• identify common differentiation errors and assess the reliability of computed gradients;
• connect these tools to optimization, neural networks, scientific computing, and machine-learning software.

1. Functions of Several Variables

A function of several variables maps a vector input to an output. A scalar-valued function may be written as

\[

f:\mathbb{R}^n\rightarrow\mathbb{R},\qquad f(\mathbf{x})=f(x_1,\ldots,x_n).

\]

For example, a linear model has the form

\[

f(\mathbf{x})=\mathbf{w}^{\mathsf T}\mathbf{x}+b,

\]

where \(\mathbf{x}\in\mathbb{R}^n\), \(\mathbf{w}\in\mathbb{R}^n\), and \(b\in\mathbb{R}\). A neural-network loss is also scalar-valued, but it depends on every weight and bias in the network. A vector-valued function maps an input vector to an output vector:

\[

\mathbf{f}:\mathbb{R}^n\rightarrow\mathbb{R}^m.

\]

The distinction between input and output dimensions is essential. A derivative is not merely a number; it is a linear map whose matrix representation depends on the dimensions and on the convention used for vectors.

Partial derivatives

The partial derivative of \(f\) with respect to \(x_i\) holds all other variables fixed:

\[

\frac{\partial f}{\partial x_i}(\mathbf{x})=

\lim_{h\to 0}\frac{f(x_1,\ldots,x_i+h,\ldots,x_n)-f(\mathbf{x})}{h}.

\]

For \(f(x,y)=x^2y+\sin y\),

\[

\frac{\partial f}{\partial x}=2xy,

\qquad

\frac{\partial f}{\partial y}=x^2+\cos y.

\]

A partial derivative describes local sensitivity in one coordinate direction. It should not be confused with the total change caused by changing several variables simultaneously.

Gradient and directional derivative

The gradient of a scalar function is the vector of first partial derivatives. Using a column-vector convention,

\[

\nabla f(\mathbf{x})=

\begin{bmatrix}

\partial f/\partial x_1\\

\vdots\\

\partial f/\partial x_n

\end{bmatrix}.

\]

For a small perturbation \(\Delta\mathbf{x}\),

\[

f(\mathbf{x}+\Delta\mathbf{x})\approx f(\mathbf{x})+\nabla f(\mathbf{x})^{\mathsf T}\Delta\mathbf{x}.

\]

If \(\mathbf{u}\) is a unit vector, the directional derivative in direction \(\mathbf{u}\) is

\[

D_{\mathbf{u}}f(\mathbf{x})=\nabla f(\mathbf{x})^{\mathsf T}\mathbf{u}.

\]

The gradient points in the direction of greatest local increase, and its negative points in the direction of greatest local decrease. This fact explains the basic update used in gradient descent:

\[

\mathbf{x}_{t+1}=\mathbf{x}_t-\eta\nabla f(\mathbf{x}_t),

\]

where \(\eta>0\) is the learning rate.

Jacobian matrices

For \(\mathbf{f}=(f_1,\ldots,f_m)^{\mathsf T}:\mathbb{R}^n\to\mathbb{R}^m\), the Jacobian is the \(m\times n\) matrix

\[

J_{\mathbf f}(\mathbf{x})=

\begin{bmatrix}

\partial f_1/\partial x_1 & \cdots & \partial f_1/\partial x_n\\

\vdots & & \vdots\\

\partial f_m/\partial x_1 & \cdots & \partial f_m/\partial x_n

\end{bmatrix}.

\]

The Jacobian is the best local linear approximation:

\[

\mathbf f(\mathbf{x}+\Delta\mathbf{x})

\approx \mathbf f(\mathbf{x})+J_{\mathbf f}(\mathbf{x})\Delta\mathbf{x}.

\]

For \(\mathbf f(x,y)=(x^2+y,\,xy)^{\mathsf T}\),

\[

J_{\mathbf f}(x,y)=

\begin{bmatrix}2x&1\\y&x\end{bmatrix}.

\]

In applications, Jacobians describe sensitivity of multiple outputs to multiple inputs. Explicitly storing a large Jacobian is often unnecessary; automatic differentiation can compute Jacobian-vector or vector-Jacobian products more efficiently.

2. Second-Order Structure and Taylor Approximation

The Hessian of a twice-differentiable scalar function is the matrix of second partial derivatives:

\[

H_f(\mathbf{x})=\nabla^2 f(\mathbf{x}),

\qquad

[H_f]_{ij}=\frac{\partial^2 f}{\partial x_i\partial x_j}.

\]

Under suitable continuity conditions, mixed partial derivatives agree, so the Hessian is symmetric. For

\[

f(x,y)=x^2+3xy+4y^2,

\]

\[

\nabla f=\begin{bmatrix}2x+3y\\3x+8y\end{bmatrix},

\qquad

H_f=\begin{bmatrix}2&3\\3&8\end{bmatrix}.

\]

The Hessian describes curvature. At a stationary point, where \(\nabla f=0\), a positive-definite Hessian indicates a strict local minimum, a negative-definite Hessian a strict local maximum, and an indefinite Hessian a saddle point. In high-dimensional machine learning, Hessians are often too large to form explicitly, but Hessian-vector products remain useful in second-order optimization and sensitivity analysis.

The second-order Taylor approximation around \(\mathbf{x}_0\) is

\[

f(\mathbf{x}_0+\Delta\mathbf{x})

\approx f(\mathbf{x}_0)+\nabla f(\mathbf{x}_0)^{\mathsf T}\Delta\mathbf{x}

+\frac12\Delta\mathbf{x}^{\mathsf T}H_f(\mathbf{x}_0)\Delta\mathbf{x}.

\]

The first-order term captures slope; the quadratic term captures curvature. Taylor models support local optimization, error analysis, uncertainty approximations, and the derivation of Newton-type methods. Their validity is local: a good approximation near \(\mathbf{x}_0\) may be poor at a distant point.

3. The Multivariable Chain Rule

Many models are compositions. Suppose

\[

\mathbf{z}=g(\mathbf{x}),\qquad \mathbf{y}=f(\mathbf{z}),

\]

where \(g:\mathbb{R}^n\to\mathbb{R}^k\) and \(f:\mathbb{R}^k\to\mathbb{R}^m\). Then

\[

J_{f\circ g}(\mathbf{x})=J_f(g(\mathbf{x}))J_g(\mathbf{x}).

\]

The order matters. The inner Jacobian acts first on a perturbation, and the outer Jacobian acts afterward. For a scalar loss \(L(\mathbf y)\), the differential notation is especially useful:

\[

dL=\nabla_{\mathbf y}L^{\mathsf T}d\mathbf y,

\qquad d\mathbf y=J_g(\mathbf x)d\mathbf x.

\]

Thus,

\[

\nabla_{\mathbf x}L=J_g(\mathbf x)^{\mathsf T}\nabla_{\mathbf y}L.

\]

The transpose appears because gradients are represented as column vectors. This relation is the mathematical core of reverse-mode differentiation.

Worked example: a two-layer computation

Consider

\[

z=wx+b,

\qquad a=\sigma(z),

\qquad L=\frac12(a-y)^2,

\]

where \(\sigma(z)=1/(1+e^{-z})\). Applying the chain rule,

\[

\frac{\partial L}{\partial a}=a-y,

\qquad

\frac{\partial a}{\partial z}=a(1-a),

\]

so

\[

\frac{\partial L}{\partial z}=(a-y)a(1-a).

\]

Finally,

\[

\frac{\partial L}{\partial w}=\frac{\partial L}{\partial z}x,

\qquad

\frac{\partial L}{\partial b}=\frac{\partial L}{\partial z}.

\]

The forward pass computes \(z\) and \(a\); the backward pass reuses them to compute parameter sensitivities.

4. Differentiation Under Constraints

Optimization often imposes conditions such as conservation laws, normalization, or resource limits. For a problem

\[

\text{minimize } f(\mathbf{x})\quad\text{subject to }g(\mathbf{x})=0,

\]

we form the Lagrangian

\[

\mathcal L(\mathbf{x},\lambda)=f(\mathbf{x})+\lambda g(\mathbf{x}).

\]

At a regular constrained stationary point,

\[

\nabla f(\mathbf{x}^)+\lambda^\nabla g(\mathbf{x}^*)=0,

\qquad g(\mathbf{x}^*)=0.

\]

Geometrically, the objective gradient is normal to the constraint surface at the optimum. With multiple constraints, multipliers form a vector and the constraint Jacobian replaces the single gradient. These conditions are necessary under regularity assumptions, not automatically sufficient for a global optimum.

For example, minimizing \(x^2+y^2\) subject to \(x+y=1\) gives

\[

2x+\lambda=0,\quad 2y+\lambda=0,\quad x+y=1.

\]

Hence \(x=y=1/2\). In machine learning, constrained differentiation appears in normalization, fairness constraints, adversarial optimization, optimal control, and differentiable physical models.

5. From Derivatives to Backpropagation

A computational graph represents a function as elementary operations: addition, multiplication, matrix multiplication, exponentiation, and nonlinear activations. Each node stores a value during the forward pass and a local derivative rule. Backpropagation applies the chain rule from the output toward the inputs.

For a scalar loss \(L\) and an intermediate variable \(v\), define the adjoint or upstream derivative as \(\bar v=\partial L/\partial v\). If \(v=\phi(u)\), then

\[

\bar u=\bar v\frac{\partial v}{\partial u}.

\]

If a variable influences the loss through several paths, the contributions are added. This addition rule is crucial in networks with branching or skip connections.

For a dense layer,

\[

\mathbf z=W\mathbf x+\mathbf b,

\qquad \mathbf a=\phi(\mathbf z),

\]

with upstream gradient \(\bar{\mathbf z}=\partial L/\partial\mathbf z\), the parameter gradients are

\[

\frac{\partial L}{\partial W}=\bar{\mathbf z}\mathbf x^{\mathsf T},

\qquad

\frac{\partial L}{\partial\mathbf b}=\bar{\mathbf z},

\qquad

\frac{\partial L}{\partial\mathbf x}=W^{\mathsf T}\bar{\mathbf z}.

\]

These formulas explain why backpropagation is computationally efficient: it computes all parameter gradients with a cost comparable to a small multiple of the forward computation, rather than perturbing every parameter separately.

6. Automatic Differentiation

Automatic differentiation (AD) evaluates derivatives by decomposing a program into elementary operations and applying the chain rule systematically. It is not the same as symbolic differentiation and not the same as finite differences.

Symbolic differentiation manipulates expressions to produce another expression. It can yield exact formulas but may cause expression growth. Finite differences approximate derivatives using function evaluations, for example,

\[

f'(x)\approx\frac{f(x+h)-f(x)}{h}.

\]

They are simple but affected by truncation and floating-point cancellation errors. AD computes derivatives to the accuracy of the underlying arithmetic, while avoiding symbolic expression expansion.

Forward mode and reverse mode

Forward-mode AD propagates a value and its derivative from inputs toward outputs. It is efficient when the number of inputs is small relative to the number of outputs. Reverse-mode AD propagates sensitivities backward from outputs to inputs. It is efficient when there are many inputs and a small number of outputs, especially a scalar loss with many parameters.

Neural-network training typically uses reverse mode. A useful computational interpretation is:

• forward mode computes Jacobian-vector products, \(J\mathbf v\);
• reverse mode computes vector-Jacobian products, \(\mathbf u^{\mathsf T}J\).

Modern systems such as PyTorch, JAX, and TensorFlow construct or trace computational graphs and provide gradient transformations. They can also compose modes to obtain Jacobians, Hessian-vector products, and higher-order derivatives.

Gradient checking

Automatic differentiation implementations still require testing. A common diagnostic compares an AD gradient with a central finite-difference estimate:

\[

\frac{f(\mathbf{x}+h\mathbf e_i)-f(\mathbf{x}-h\mathbf e_i)}{2h}.

\]

Agreement should be assessed within an appropriate tolerance and typically performed on small, well-scaled examples. Gradient checks can fail because of nondifferentiable operations, stochastic layers, unsuitable step sizes, floating-point precision, or an incorrect reference implementation.

7. Applications in Machine Learning and Industry

Gradients drive parameter estimation in linear regression, logistic regression, neural networks, matrix factorization, and reinforcement learning. Jacobians are important in uncertainty propagation, normalizing flows, computer vision, robotics, and sensitivity analysis. Hessians and curvature approximations support trust-region methods, Newton methods, meta-learning, and analysis of optimization landscapes.

In robotics, the Jacobian maps joint velocities to end-effector velocity and helps solve inverse-kinematics problems. In probabilistic modeling, the determinant of a transformation Jacobian appears in change-of-variables formulas. In scientific machine learning, derivatives enforce differential equations and physical constraints. In operations and finance, differentiable simulators permit optimization through pricing, control, and resource-allocation models.

Professional practice requires attention to dimensions, batch axes, parameter sharing, and numerical behavior. A gradient may be mathematically correct yet unusable if it is poorly scaled, contains NaNs, or is computed through an unstable implementation. Log-sum-exp transformations, stable softmax formulas, normalization, gradient clipping, and suitable parameterizations are practical responses to these issues.

Illustrative Case Study: Training a Small Neural Classifier

Background

A research team develops a two-layer classifier for identifying whether a sensor record belongs to class 0 or class 1. Each record has four standardized features. The model is

\[

\mathbf h=\operatorname{ReLU}(W_1\mathbf x+\mathbf b_1),

\qquad

\hat y=\sigma(\mathbf w_2^{\mathsf T}\mathbf h+b_2),

\]

with binary cross-entropy loss.

Situation and problem

The initial training run reports decreasing loss, but validation performance is inconsistent. The team must determine whether the problem is model capacity, an incorrect gradient, or unstable numerical computation.

Analysis and approach

The team first writes the graph explicitly. For one example, the forward variables are \(\mathbf z_1=W_1\mathbf x+\mathbf b_1\), \(\mathbf h=\operatorname{ReLU}(\mathbf z_1)\), \(z_2=\mathbf w_2^{\mathsf T}\mathbf h+b_2\), and \(\hat y=\sigma(z_2)\). For binary cross-entropy, the output derivative simplifies to

\[

\frac{\partial L}{\partial z_2}=\hat y-y.

\]

The backward pass then gives

\[

\frac{\partial L}{\partial \mathbf w_2}=(\hat y-y)\mathbf h,

\qquad

\frac{\partial L}{\partial b_2}=\hat y-y,

\]

\[

\frac{\partial L}{\partial \mathbf h}=(\hat y-y)\mathbf w_2,

\qquad

\frac{\partial L}{\partial \mathbf z_1}=\frac{\partial L}{\partial\mathbf h}\odot\mathbf 1_{\mathbf z_1>0}.

\]

Finally, the first-layer gradients are

\[

\frac{\partial L}{\partial W_1}=\frac{\partial L}{\partial\mathbf z_1}\mathbf x^{\mathsf T},

\qquad

\frac{\partial L}{\partial\mathbf b_1}=\frac{\partial L}{\partial\mathbf z_1}.

\]

The implementation is tested on a single record using both reverse-mode AD and central differences. The results agree except at units exactly equal to zero, where ReLU is nondifferentiable. The team adopts a documented subgradient convention, checks batch aggregation, and replaces a naive probability calculation with a numerically stable logits-based loss. It also inspects gradient norms and confirms that feature standardization is applied identically during training and validation.

Outcome

The primary error is found to be an incorrect transpose in the first-layer gradient. After correction, gradient checks pass away from ReLU kink points, training becomes stable, and validation performance improves. The investigation also establishes a repeatable testing procedure rather than relying solely on a decreasing training loss.

Learning points

The case illustrates that differentiation is simultaneously a mathematical, software, and systems concern. Explicit computational graphs reveal dependency errors; dimension checks expose transpose mistakes; AD reduces implementation effort but does not eliminate the need for validation; and numerical stability is part of derivative correctness in practical machine learning.

Limitations, Risks, and Common Mistakes

Several errors recur in multivariable differentiation. Treating a vector as a scalar can hide missing dimensions. Reversing Jacobian multiplication violates the order of composition. Confusing a gradient with a Jacobian can produce incorrect transposes. Forgetting to sum gradients over repeated uses or batch examples leads to silent errors. A derivative at a nondifferentiable point may not exist, although software often selects a subgradient.

Vanishing and exploding gradients arise when products of Jacobians have very small or very large norms. Saturating activations, poorly scaled inputs, excessive depth, and ill-conditioned objectives contribute to these problems. Gradient descent can also fail because the learning rate is too large, the objective is nonconvex, or the gradient is noisy.

AD has limitations. It consumes memory, particularly in reverse mode because intermediate values must be retained. Control flow, mutation, discontinuities, custom numerical kernels, and implicit algorithms may require special treatment. A computed derivative is not automatically a useful derivative: it may describe an unstable or poorly conditioned model.

Professional and Ethical Considerations

Differentiable systems can optimize decisions affecting people, such as lending, medical triage, employment, or allocation of public resources. A technically accurate gradient may still optimize an inappropriate objective. Practitioners should document objectives, constraints, data transformations, parameterizations, and nondifferentiability conventions. They should test sensitivity to perturbations and monitor whether optimization exploits spurious correlations or violates fairness and safety requirements.

In regulated or high-impact settings, reproducible gradient calculations, versioned code, audit trails, and independent validation are important. Automatic differentiation improves consistency, but it does not replace domain review, statistical validation, or governance.

Emerging Directions and Best Practices

Differentiable programming increasingly combines machine-learning components with simulators, solvers, databases, and control systems. Neural ordinary differential equations, differentiable rendering, physics-informed learning, and implicit layers require derivatives through iterative or constrained computations. Research also investigates memory-efficient AD, compiler transformations, mixed-mode differentiation, and reliable higher-order derivatives.

Effective practice includes:

• write the mathematical computation graph before coding;
• annotate tensor shapes and distinguish batch, feature, and parameter axes;
• use stable elementary functions and well-scaled inputs;
• verify small examples with analytical results and finite differences;
• inspect gradient norms, loss scales, and exceptional values;
• treat nondifferentiable operations explicitly;
• retain unit tests for custom backward rules;
• use reverse mode for scalar-loss training and choose modes according to input-output dimensions;
• interpret optimization results in relation to the original objective and constraints.

Chapter Summary

Multivariable calculus extends the derivative from one-dimensional rates of change to local linear and quadratic descriptions of functions with many variables. Partial derivatives measure coordinate-wise sensitivity; gradients describe scalar-function change and steepest directions; Jacobians represent derivatives of vector-valued functions; and Hessians describe curvature. Taylor approximations provide local models that support optimization and error analysis.

The multivariable chain rule explains how derivatives propagate through compositions. Its reverse application is backpropagation, in which local derivatives are combined from outputs toward inputs. Automatic differentiation implements this process over computational graphs, with forward mode producing Jacobian-vector products and reverse mode producing vector-Jacobian products. These methods are central to modern machine learning, but their use requires attention to dimensions, numerical stability, nonsmooth operations, memory, validation, and the appropriateness of the objective being optimized.

Key Terms

Partial derivative: Derivative with respect to one variable while holding other variables fixed.

Gradient: Column vector of first partial derivatives of a scalar-valued function.

Directional derivative: Rate of change in a specified direction.

Jacobian: Matrix of first derivatives of a vector-valued function.

Hessian: Matrix of second partial derivatives of a scalar-valued function.

Differential: Linear first-order change in a function caused by an input perturbation.

Taylor approximation: Local polynomial approximation based on derivatives at a reference point.

Chain rule: Rule for differentiating a composition of functions.

Lagrangian: Objective augmented with weighted constraint functions.

Constraint qualification: Regularity condition supporting constrained-stationarity results.

Computational graph: Directed representation of elementary operations forming a computation.

Backpropagation: Reverse application of the chain rule through a computational graph.

Adjoint: Upstream derivative of a loss with respect to an intermediate variable.

Automatic differentiation: Algorithmic evaluation of derivatives using elementary-operation rules.

Forward mode: AD method that propagates derivatives from inputs to outputs.

Reverse mode: AD method that propagates sensitivities from outputs to inputs.

Jacobian-vector product: Product \(J\mathbf v\), commonly computed efficiently by forward mode.

Vector-Jacobian product: Product \(\mathbf u^{\mathsf T}J\), commonly computed efficiently by reverse mode.

Subgradient: Generalized gradient selected at certain nonsmooth points.

Gradient checking: Comparison of computed derivatives with an independent numerical approximation.

Review Questions

Short Answer Questions

1. Define a gradient and explain its geometric meaning.

2. What information does a Hessian provide that a gradient does not?

3. State the Jacobian chain rule for two vector-valued functions.

4. Distinguish automatic differentiation from finite-difference differentiation.

5. Why is reverse-mode differentiation suitable for training a neural network with a scalar loss?

Long Answer Questions

1. Derive the first- and second-order Taylor approximations for a scalar function of several variables and explain their uses in optimization.

2. Explain partial derivatives, gradients, directional derivatives, and Jacobians, including their dimensions and interpretations.

3. Derive the gradients of a dense neural-network layer \(\mathbf z=W\mathbf x+\mathbf b\) from an upstream gradient.

4. Explain Lagrange multipliers geometrically and algebraically for equality-constrained optimization.

5. Compare symbolic differentiation, finite differences, forward-mode AD, and reverse-mode AD in terms of computation and use cases.

Analytical / Application Questions

1. For \(f(x,y)=x^2y+\sin y\), compute the gradient and Hessian, then use the gradient to estimate the change in \(f\) caused by a small perturbation.

2. A model maps \(\mathbf x\in\mathbb R^5\) to \(\mathbf y\in\mathbb R^3\). Determine the dimensions of its Jacobian and explain whether forward or reverse mode is likely to be more efficient for one input-output evaluation.

3. A gradient check disagrees with an automatic-differentiation result near a ReLU unit whose preactivation is zero. Analyze possible reasons and propose a responsible validation procedure.

22. Multiple Choice Questions with Answers

Question 1

For a scalar function f: R^n -> R, which object contains its first partial derivatives?

A. The gradient

B. The Hessian

C. The constraint Jacobian only

D. The Taylor remainder

Correct Answer: A. The gradient

Question 2

If f: R^n -> R^m, what is the dimension of its Jacobian under the convention used in this chapter?

A. n x n

B. m x n

C. n x m

D. m x m

Correct Answer: B. m x n

Question 3

For a composition y=f(g(x)), which expression gives the Jacobian of the composition?

A. J_g J_f

B. J_f + J_g

C. J_f(g(x)) J_g(x)

D. J_f(g(x))^T J_g(x)^T

Correct Answer: C. J_f(g(x)) J_g(x)

Question 4

Which statement about the gradient is correct?

A. It always points toward the nearest minimum

B. It points in the direction of greatest local increase

C. It contains second-order curvature information

D. It exists only for vector-valued functions

Correct Answer: B. It points in the direction of greatest local increase

Question 5

Why is reverse-mode automatic differentiation efficient for neural-network training?

A. It avoids all matrix operations

B. It is efficient when there is one scalar output and many inputs

C. It requires no stored intermediate values

D. It computes only one parameter derivative

Correct Answer: B. It is efficient when there is one scalar output and many inputs

Question 6

For z = W x + b and upstream gradient dz, which is the gradient with respect to W?

A. x dz^T

B. dz x^T

C. W^T dz

D. dz + x

Correct Answer: B. dz x^T

Question 7

At a constrained optimum satisfying g(x)=0, the Lagrange multiplier condition states that:

A. The objective gradient must be zero in every direction

B. The objective gradient is proportional to the constraint gradient

C. The Hessian must be the identity matrix

D. The constraint must be linear

Correct Answer: B. The objective gradient is proportional to the constraint gradient

Question 8

Which method is most directly affected by subtractive cancellation when the step size is extremely small?

A. Finite differences

B. Reverse-mode AD

C. Symbolic simplification

D. The Lagrange multiplier method

Correct Answer: A. Finite differences

Question 9

A Hessian that is indefinite at a stationary point generally indicates:

A. A strict local minimum

B. A strict local maximum

C. A saddle-point structure

D. That the function is linear everywhere

Correct Answer: C. A saddle-point structure

Question 10

In reverse-mode AD, if an intermediate variable influences a loss through multiple paths, its gradient contributions are:

A. Multiplied together

B. Ignored except for the shortest path

C. Added together

D. Replaced by the Hessian

Correct Answer: C. Added together