9. Mathematical Optimization for Machine Learning

Introduction and Context

Machine learning systems are built by selecting parameters that make a model perform well according to a specified objective. In linear regression, the objective may be the sum of squared prediction errors; in classification, it may be cross-entropy loss; in deep learning, it is usually a non-convex function of millions or billions of parameters. The process of finding suitable parameter values is an optimization problem.

Optimization provides the mathematical language and computational machinery for this process. It explains how a model objective is formulated, how feasible solutions are characterized, why some algorithms converge, why others may be unstable, and how computational constraints influence practical choices. Although optimization is often introduced as a search for a minimum, machine learning applications require a broader view. The objective may be non-convex, data may arrive in streams or mini-batches, constraints may encode fairness or resource limits, and the solution may need to generalize rather than merely fit the training data.

This chapter develops the central ideas of unconstrained and constrained optimization for machine learning. It connects derivatives, convexity, Lagrange multipliers, Karush–Kuhn–Tucker conditions, gradient-based methods, stochastic optimization, momentum, and adaptive learning rates. It also examines practical failure modes and shows how optimization decisions affect model training in professional settings.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Formulate machine-learning training tasks as unconstrained or constrained optimization problems.

2. Interpret gradients, Hessians, stationary points, and second-order conditions.

3. Explain convex sets, convex functions, strong convexity, and their consequences for global optimality.

4. Apply Lagrange multipliers and KKT conditions to constrained problems.

5. Compare batch gradient descent, stochastic gradient descent, mini-batch methods, and second-order approaches.

6. Explain the roles of momentum, learning-rate schedules, and adaptive optimizers such as AdaGrad, RMSProp, and Adam.

7. Diagnose optimization difficulties involving scaling, ill-conditioning, saddle points, noisy gradients, and overfitting.

8. Select optimization methods responsibly for practical machine-learning systems.

9.1 Optimization Problems in Machine Learning

A general optimization problem can be written as

\[

\min_{\theta \in \mathbb{R}^d} f(\theta),

\]

where \(\theta\) is a vector of decision variables and \(f\) is an objective function. In supervised learning, \(\theta\) represents model parameters and the objective commonly includes empirical risk and regularization:

\[

f(\theta)=\frac{1}{n}\sum_{i=1}^{n}\ell(h_\theta(x_i),y_i)+\lambda R(\theta).

\]

Here, \(\ell\) is a loss function, \(h_\theta\) is the model, \(R\) is a regularizer, and \(\lambda\) controls the regularization strength. The optimization formulation makes explicit what is minimized, which variables are adjustable, and what restrictions apply.

An unconstrained problem has no explicit restrictions beyond the domain of the objective. A constrained problem includes equality constraints, inequality constraints, or both:

\[

\begin{aligned}

\min_\theta \quad & f(\theta)\\

\text{subject to}\quad & g_j(\theta)\leq 0,\quad j=1,\ldots,m,\\

& h_k(\theta)=0,\quad k=1,\ldots,p.

\end{aligned}

\]

Constraints can represent a probability vector summing to one, a non-negative parameter, a budget, a safety threshold, a fairness requirement, or a physical limitation. Regularization is not always a constraint, although many regularized objectives have an equivalent constrained interpretation. For example, minimizing loss subject to \(\|\theta\|_2^2\leq c\) is closely related to minimizing loss plus an \(L_2\) penalty under suitable conditions.

Essential terminology

A feasible point satisfies all constraints. The feasible region is the set of all feasible points. A global minimizer has no feasible point with a lower objective value. A local minimizer is optimal only within a neighborhood. A stationary point has zero gradient, although it need not be a minimum. A descent direction is a direction in which the objective decreases locally. An algorithm is said to converge when its iterates, objective values, or gradients approach a specified limiting condition.

The distinction between optimization and generalization is important. An optimizer may find a parameter vector with low training loss, but low training loss does not guarantee good performance on unseen data. Regularization, validation, early stopping, data quality, and model architecture influence generalization in addition to the optimization procedure.

9.2 Local Geometry: Gradients, Hessians, and Curvature

For a differentiable scalar function \(f(\theta)\), the gradient is

\[

\nabla f(\theta)=\left[\frac{\partial f}{\partial \theta_1},\ldots,\frac{\partial f}{\partial \theta_d}\right]^T.

\]

The gradient points in the direction of steepest local increase under the Euclidean norm. Consequently, \(-\nabla f(\theta)\) is a locally steepest descent direction. The first-order Taylor approximation is

\[

f(\theta+\Delta)\approx f(\theta)+\nabla f(\theta)^T\Delta.

\]

This expression motivates gradient methods: choose a step in a direction having a negative inner product with the gradient.

The Hessian matrix contains second derivatives:

\[

H(\theta)=\nabla^2f(\theta).

\]

It describes local curvature. If the Hessian is positive definite at a stationary point, the point is a strict local minimum. If it is negative definite, the point is a strict local maximum. If it has both positive and negative eigenvalues, the point is a saddle point. A positive semidefinite Hessian gives a necessary condition for a local minimum but may not be sufficient without additional analysis.

Curvature affects optimization speed. In a narrow valley, the gradient may point substantially across the valley rather than directly toward its minimum. A single learning rate then tends to produce oscillation in high-curvature directions and slow movement in low-curvature directions. The ratio between the largest and smallest relevant curvature values is related to the condition number. Poor conditioning is a major reason that scaling features and using preconditioning or adaptive methods can improve training.

9.3 Convexity and Global Optimality

A set \(C\) is convex if, for every \(u,v\in C\) and \(t\in[0,1]\), the point \(tu+(1-t)v\) also belongs to \(C\). A differentiable function is convex on a convex domain if

\[

f(v)\geq f(u)+\nabla f(u)^T(v-u)

\]

for all \(u,v\) in the domain. Geometrically, the graph lies above every tangent plane. If the function is twice differentiable, a sufficient condition for convexity is

\[

\nabla^2 f(\theta)\succeq 0

\]

throughout the domain. Strong convexity requires a positive curvature lower bound, such as \(\nabla^2 f(\theta)\succeq \mu I\) for some \(\mu>0\).

Convexity is valuable because every local minimum of a convex function is global. If a differentiable convex function has a point \(\theta^\) satisfying \(\nabla f(\theta^)=0\), then \(\theta^*\) is a global minimizer. Strong convexity also gives uniqueness of the minimizer and supports stronger convergence guarantees.

Examples include least-squares regression with a linear model, logistic regression with the standard log-loss, and many regularized generalized linear models. In contrast, neural-network training is generally non-convex because compositions of nonlinear layers create complicated objective surfaces. Non-convexity does not imply that optimization is impossible; it means that global guarantees are more limited and initialization, architecture, stochasticity, and algorithm design become important.

A useful distinction is between a global optimum, a local optimum, and a stationary point. In high-dimensional non-convex problems, saddle points and flat regions may be more consequential than poor local minima. Modern stochastic methods can sometimes escape certain strict saddle points because random gradient noise perturbs the iterates, but this is not a universal guarantee.

9.4 Constrained Optimization: Lagrange Multipliers and KKT Conditions

Suppose the problem has an equality constraint \(h(\theta)=0\). The Lagrangian is

\[

\mathcal{L}(\theta,\nu)=f(\theta)+\nu h(\theta),

\]

where \(\nu\) is a Lagrange multiplier. At a regular constrained optimum, the objective gradient need not be zero. Instead, it can be balanced by the normal direction of the constraint:

\[

\nabla_\theta\mathcal{L}(\theta^,\nu^)=0,

\qquad h(\theta^*)=0.

\]

For multiple equality and inequality constraints, use

\[

\mathcal{L}(\theta,\nu,\mu)=f(\theta)+\sum_{k=1}^{p}\nu_kh_k(\theta)+\sum_{j=1}^{m}\mu_jg_j(\theta).

\]

For the convention \(g_j(\theta)\leq0\), the Karush–Kuhn–Tucker conditions are:

• Stationarity: \(\nabla f(\theta^)+\sum_k\nu_k^\nabla h_k(\theta^)+\sum_j\mu_j^\nabla g_j(\theta^*)=0\).
• Primal feasibility: \(h_k(\theta^)=0\) and \(g_j(\theta^)\leq0\).
• Dual feasibility: \(\mu_j^*\geq0\).
• Complementary slackness: \(\mu_j^g_j(\theta^)=0\) for every inequality constraint.

Complementary slackness states that either a constraint is active, meaning \(g_j(\theta^*)=0\), or its multiplier is zero. Under suitable constraint qualifications, KKT conditions are necessary for a local optimum. In convex optimization, if the objective and inequality constraints are convex, equality constraints are affine, and an appropriate regularity condition such as Slater’s condition holds, KKT conditions are also sufficient for global optimality.

The multipliers have an economic and operational interpretation. A multiplier associated with a resource constraint often approximates the change in optimal objective value resulting from a small relaxation of that constraint. This sensitivity interpretation is useful in capacity planning and model governance.

A simple constrained example

Consider

\[

\min_{x,y}\; x^2+y^2

\quad\text{subject to}\quad x+y=1.

\]

The Lagrangian is \(\mathcal{L}=x^2+y^2+\nu(x+y-1)\). Stationarity gives \(2x+\nu=0\) and \(2y+\nu=0\), so \(x=y\). Feasibility then gives \(x=y=1/2\). The closest point to the origin on the line \(x+y=1\) is therefore \((1/2,1/2)\). The example illustrates why the unconstrained gradient condition is replaced by a balance between objective and constraint gradients.

9.5 Gradient-Based Algorithms

Batch gradient descent

For an unconstrained differentiable objective, gradient descent updates parameters according to

\[

\theta_{t+1}=\theta_t-\eta_t\nabla f(\theta_t),

\]

where \(\eta_t>0\) is the learning rate. If the objective has Lipschitz-continuous gradient with constant \(L\), sufficiently small fixed steps, commonly bounded by a quantity related to \(1/L\), provide convergence for convex settings. In practice, the appropriate step depends on scale, curvature, architecture, and data.

Batch gradient descent computes the gradient using the complete training set. It provides a relatively accurate direction but can be expensive for very large datasets. It is useful when the dataset is small, deterministic reproducibility is important, or objective evaluation is inexpensive.

Stochastic and mini-batch gradient descent

If the empirical objective is an average of per-example losses, a randomly sampled example produces an unbiased or approximately unbiased gradient estimate:

\[

\widehat{g}_t=\nabla_\theta\ell(h_{\theta_t}(x_{i_t}),y_{i_t}).

\]

Stochastic gradient descent (SGD) uses

\[

\theta_{t+1}=\theta_t-\eta_t\widehat{g}_t.

\]

Mini-batch methods average gradients over a batch of examples. They balance computational efficiency, parallel hardware utilization, and variance reduction. Larger batches generally produce less noisy estimates but may require more memory and can alter the optimization dynamics. Gradient accumulation can emulate a larger batch when memory is limited.

The noise in SGD is not merely a defect. It can help explore non-convex landscapes and sometimes favors solutions that generalize well, although the relationship depends on batch size, learning rate, architecture, and data. A decaying learning rate is often used when convergence to a stable neighborhood or solution is required. In deep learning, schedules such as step decay, cosine decay, and warm-up are commonly used, but their suitability depends on the training regime.

Projected and proximal methods

When parameters must remain in a feasible set \(C\), projected gradient descent first takes a gradient step and then projects back:

\[

\theta_{t+1}=\Pi_C(\theta_t-\eta_t\nabla f(\theta_t)),

\]

where \(\Pi_C(z)\) is the closest point in \(C\) under the selected norm. This is useful for non-negative parameters, norm-bounded models, and simplex constraints.

Proximal methods handle objectives of the form \(f(\theta)+r(\theta)\), where \(r\) may be non-smooth. The proximal update is

\[

\theta_{t+1}=\operatorname{prox}_{\eta r}(\theta_t-\eta\nabla f(\theta_t)).

\]

For an \(L_1\) penalty, the proximal operator is soft-thresholding, which can produce exact zeros and hence sparse solutions. This is often preferable to treating a non-smooth penalty with an ordinary derivative-based update.

Second-order and quasi-Newton methods

Newton’s method uses curvature:

\[

\theta_{t+1}=\theta_t-[\nabla^2f(\theta_t)]^{-1}\nabla f(\theta_t).

\]

Near a well-behaved optimum, it can converge rapidly, but computing and inverting a large Hessian is impractical for modern neural networks. Quasi-Newton methods, such as limited-memory BFGS, approximate inverse curvature using a history of gradients and steps. They can be highly effective for medium-scale smooth problems, while SGD variants are usually preferred for massive stochastic objectives.

9.6 Momentum and Adaptive Learning Rates

Plain gradient descent may zigzag in elongated valleys. Momentum introduces a velocity that accumulates past gradients. One common form is

\[

v_{t+1}=\beta v_t+(1-\beta)g_t,

\qquad

\theta_{t+1}=\theta_t-\eta v_{t+1},

\]

where \(g_t\) is the current gradient and \(0\leq\beta<1\). Momentum accelerates consistent movement and reduces oscillation caused by rapidly changing gradient directions.

Nesterov accelerated gradient evaluates the gradient at a look-ahead position, often improving the anticipation of future movement. Both methods can be understood as ways of filtering gradient information over time rather than reacting independently to every mini-batch.

Adaptive methods maintain coordinate-specific scale estimates. AdaGrad accumulates squared gradients:

\[

r_t=r_{t-1}+g_t\odot g_t,

\qquad

\theta_{t+1}=\theta_t-\eta\frac{g_t}{\sqrt{r_t}+\epsilon}.

\]

It gives relatively larger effective steps to coordinates with small historical gradients, but the accumulated denominator can grow continually and make learning stop prematurely.

RMSProp replaces the unbounded sum with an exponential moving average:

\[

r_t=\rho r_{t-1}+(1-\rho)g_t\odot g_t,

\qquad

\theta_{t+1}=\theta_t-\eta\frac{g_t}{\sqrt{r_t}+\epsilon}.

\]

Adam combines first-moment momentum and second-moment scaling. With

\[

m_t=\beta_1m_{t-1}+(1-\beta_1)g_t,

\quad

v_t=\beta_2v_{t-1}+(1-\beta_2)g_t^2,

\]

bias-corrected estimates are formed as \(\widehat m_t=m_t/(1-\beta_1^t)\) and \(\widehat v_t=v_t/(1-\beta_2^t)\), followed by

\[

\theta_{t+1}=\theta_t-\eta\frac{\widehat m_t}{\sqrt{\widehat v_t}+\epsilon}.

\]

Adam is often convenient for sparse gradients, noisy objectives, and rapid experimentation. It is not automatically superior: its adaptive scaling, weight decay implementation, and learning-rate schedule can affect generalization. Decoupled weight decay, as in AdamW-style approaches, separates parameter shrinkage from the adaptive gradient update and is often a better-controlled regularization mechanism.

9.7 A Practical Optimization Workflow

A reliable optimization process begins before selecting an optimizer. First, express the objective precisely, identify parameter domains, and decide whether constraints are essential or can be represented by reparameterization. For example, a softmax transformation guarantees that a vector is non-negative and sums to one, while an exponential or softplus transformation can enforce positivity.

Next, inspect data and parameter scales. Standardizing input features often improves conditioning in linear and neural models. Check that the loss and gradient are numerically finite. Gradient checking with finite differences can help validate an implementation on a small problem, although it is unsuitable as a routine method for large neural networks.

Choose a baseline optimizer and establish a reproducible evaluation protocol. Monitor training loss, validation loss, gradient norms, parameter norms, learning rates, and relevant task metrics. A decreasing training loss accompanied by worsening validation performance indicates overfitting rather than successful model development. Exploding losses or gradients may signal an excessive learning rate, unstable architecture, poor data scaling, or numerical overflow. Little movement may indicate a learning rate that is too small, saturated activations, excessive regularization, or vanishing gradients.

Learning-rate selection is often more consequential than choosing between closely related optimizers. A short range test, logarithmic learning-rate search, or principled schedule can be used. Early stopping should be based on validation evidence, with a checkpoint of the best validation model rather than simply the final iterate. For constrained problems, feasibility violations must be monitored separately from objective values.

9.8 Applied Case: Constrained Credit-Risk Model Training

Background

The following is an illustrative case involving a financial institution developing a credit-risk model. The institution wants a model that predicts probability of default while satisfying operational and governance requirements.

Situation and problem

A logistic model produces a default probability \(p_i=\sigma(x_i^T\theta)\). The training objective is regularized cross-entropy. The institution imposes three requirements: coefficients for selected protected or legally sensitive variables must be fixed at zero; predicted probabilities must satisfy a calibration-related constraint on a monitored group; and the model should not exceed a specified coefficient norm to limit instability.

These requirements create a constrained optimization problem. A naive unconstrained optimizer may reduce loss while violating governance requirements. Simply deleting variables does not address the calibration or norm constraints.

Analysis and approach

The modeling team first standardizes continuous features and separates training, validation, and audit datasets. It formulates equality constraints for fixed coefficients and inequality constraints for the norm and calibration requirements. Because logistic loss with suitable regularization is convex in the coefficient vector, the problem has a favorable global structure, provided the calibration constraint is represented in a convex or appropriately approximated form.

For the equality constraints, the team uses either a reduced parameterization or a Lagrangian method. For the norm restriction, projected gradient steps are considered. At each iteration, the model computes the gradient of the loss, takes a step, and projects the coefficient vector into the permitted norm set while preserving fixed coefficients. Constraint residuals and multipliers are monitored. If calibration constraints are active, their associated multipliers indicate the optimization cost of enforcing them.

The team compares projected gradient descent with a constrained solver on a reduced development sample. It verifies gradients numerically, tests several learning rates, and examines performance across demographic groups. Model selection is based not only on average validation loss but also on calibration, stability, interpretability, and compliance metrics.

Outcome

The final model has slightly higher unconstrained loss than the unconstrained baseline but satisfies the specified constraints and displays more stable audit performance. The team documents the objective, constraint definitions, optimization tolerance, data version, random seeds, and stopping rule. This documentation makes the result reproducible and reviewable.

Learning points

The case demonstrates that optimization is not merely a numerical exercise. Constraints express institutional values and legal or operational requirements. A mathematically efficient solution that violates those requirements is not an acceptable production model. It also illustrates the value of convexity, explicit feasibility monitoring, and separating model performance from compliance evidence.

9.9 Challenges, Limitations, and Professional Responsibilities

Optimization algorithms are sensitive to parameterization and scale. A mathematically equivalent formulation can have very different numerical behavior. Non-smooth objectives, discrete decisions, noisy labels, missing data, and non-differentiable operations can invalidate assumptions behind standard gradient methods.

Convergence of an optimizer does not establish that the model is correct, fair, robust, or causally meaningful. Training data may encode historical discrimination, and optimizing predictive loss can reproduce or amplify unequal error patterns. Constraints used as fairness proxies may conflict with one another or behave differently across populations. Their definitions, data requirements, and trade-offs should be documented with domain experts.

In safety-critical or regulated applications, organizations should maintain records of objective functions, data provenance, parameter settings, random seeds, software versions, stopping criteria, constraint tolerances, and validation results. Numerical reproducibility is not the same as accountability, but it is a necessary component of accountability. Privacy-preserving training, differential privacy, and federated optimization introduce additional noise and constraints, requiring explicit analysis of the resulting utility and convergence effects.

Common mistakes include confusing a zero gradient with a global minimum, using a learning rate without considering scale, applying an adaptive optimizer without tuning its schedule, ignoring validation performance, treating penalty terms as exact constraints, and failing to verify constraint feasibility. Another mistake is reporting only the final loss while concealing instability, failed runs, or sensitivity to initialization.

9.10 Current Directions and Best Practices

Current optimization research increasingly addresses large-scale distributed training, communication-efficient optimization, sharpness and generalization, normalization, constrained and multi-objective learning, and optimization under privacy or robustness requirements. Preconditioning, second-order approximations, and automated learning-rate schedules continue to improve computational efficiency. In large language and multimodal models, memory-aware optimizers and low-rank or quantized update strategies reduce the cost of parameter adaptation.

For professional practice, the following principles are especially useful:

• formulate the objective and constraints explicitly;
• scale data and inspect conditioning before extensive tuning;
• begin with a simple, reproducible baseline;
• monitor both optimization statistics and validation behavior;
• use appropriate projections or proximal operators for hard constraints and non-smooth penalties;
• tune learning rates and batch sizes jointly rather than treating them as independent;
• compare optimizer choices using fixed data splits and documented budgets;
• test sensitivity to initialization, random seeds, and perturbations;
• distinguish training success from generalization, fairness, safety, and compliance;
• preserve an auditable record of the optimization procedure.

Chapter Summary

Mathematical optimization provides the foundation for training machine-learning models. Problems may be unconstrained or may include equality and inequality restrictions. Gradients describe first-order change, while Hessians describe curvature and conditioning. Convexity guarantees that local minima are global and supports strong theoretical results, whereas neural-network objectives are generally non-convex and require practical diagnostics.

Lagrange multipliers and KKT conditions characterize constrained optima through stationarity, feasibility, dual feasibility, and complementary slackness. Gradient descent, SGD, and mini-batch methods trade computational cost against gradient noise. Momentum improves movement through poorly conditioned regions, while AdaGrad, RMSProp, and Adam adapt coordinate-wise step sizes. Projection and proximal methods are useful for explicit constraints and non-smooth regularization.

Effective optimization requires more than reducing training loss. Feature scaling, learning-rate selection, validation monitoring, numerical checks, constraint verification, reproducibility, and governance are essential. In professional AI systems, the optimization formulation must reflect technical, operational, ethical, and regulatory requirements.

Key Terms

Gradient: Vector of first partial derivatives indicating local direction of greatest increase.

Hessian: Matrix of second partial derivatives describing local curvature.

Stationary point: Point at which the gradient is zero or, in constrained settings, satisfies an appropriate stationarity condition.

Convex set: Set containing every line segment joining any two of its points.

Convex function: Function whose value lies above each of its tangent-plane approximations.

Strong convexity: Convexity with a positive lower bound on curvature.

Feasible point: Point satisfying all constraints.

Lagrangian: Function combining an objective with weighted equality and inequality constraints.

Lagrange multiplier: Variable associated with a constraint, often interpreted as a sensitivity or shadow price.

KKT conditions: Necessary, and under suitable convexity conditions sufficient, conditions for constrained optimality.

Primal feasibility: Satisfaction of the original optimization constraints.

Dual feasibility: Non-negativity of multipliers for constraints written as less than or equal to zero.

Complementary slackness: Condition that the product of each inequality multiplier and constraint value is zero.

Learning rate: Scalar controlling the size of an optimization update.

Stochastic gradient: Gradient estimate computed from a random example or mini-batch.

Momentum: Method that accumulates historical gradient information to smooth and accelerate updates.

Adaptive optimization: Optimization that adjusts effective step sizes using coordinate-wise gradient statistics.

Projection: Mapping of an infeasible point to the nearest point in a specified feasible set.

Proximal operator: Operator used to handle a regularization or penalty term, especially a non-smooth one.

Condition number: Measure, often based on curvature eigenvalues, of sensitivity and ill-conditioning.

Review Questions

Short Answer Questions

1. Distinguish between an unconstrained and a constrained optimization problem.

2. Why does a positive definite Hessian at a stationary point indicate a strict local minimum?

3. State the four groups of KKT conditions for inequality-constrained optimization.

4. What is the purpose of a learning rate in gradient descent?

5. Why can feature scaling improve optimization?

Long Answer Questions

1. Explain convexity, strong convexity, and their implications for machine-learning optimization.

2. Derive the Lagrange multiplier conditions for a single equality-constrained problem and interpret the multiplier.

3. Compare batch gradient descent, SGD, and mini-batch gradient descent in terms of computation, noise, and practical use.

4. Explain momentum, AdaGrad, RMSProp, and Adam, including their principal update mechanisms.

5. Discuss how optimization success differs from generalization and responsible model deployment.

Analytical / Application Questions

1. Formulate a regularized logistic-regression problem with a coefficient-norm constraint and identify its objective, variables, and feasible set.

2. A training loss oscillates sharply while validation loss changes slowly. Diagnose possible optimization causes and propose a systematic response.

3. A model satisfies its prediction-loss target but violates a fairness constraint. Explain how KKT-based or projected optimization could be used and what additional governance questions must be addressed.

22. Multiple Choice Questions with Answers

Question 1

Which statement best characterizes a convex differentiable optimization problem?

A. Every stationary point is necessarily a local maximum.

B. Every local minimum is also a global minimum.

C. The Hessian must be negative definite everywhere.

D. Gradient descent is guaranteed to reach the solution in one iteration.

Correct Answer: B. Every local minimum is also a global minimum.

Question 2

For constraints written as g_j(theta) <= 0, what does dual feasibility require?

A. g_j(theta) must equal zero.

B. The objective gradient must be zero independently of constraints.

C. Each inequality multiplier must be non-negative.

D. Each inequality multiplier must be negative.

Correct Answer: C. Each inequality multiplier must be non-negative.

Question 3

What is the main purpose of complementary slackness in the KKT conditions?

A. It requires every constraint to be inactive.

B. It links an inequality multiplier to whether its constraint is active.

C. It replaces primal feasibility.

D. It guarantees that the objective is non-convex.

Correct Answer: B. It links an inequality multiplier to whether its constraint is active.

Question 4

Why can mini-batch gradient descent be preferable to full-batch gradient descent for large datasets?

A. It eliminates all gradient noise.

B. It requires computation over the entire dataset at every update.

C. It balances computational efficiency, parallelism, and gradient variance.

D. It guarantees a global optimum for neural networks.

Correct Answer: C. It balances computational efficiency, parallelism, and gradient variance.

Question 5

A model oscillates across a narrow valley during training. Which intervention is most directly motivated by the chapter?

A. Increase the learning rate without testing.

B. Use momentum or improve conditioning through scaling or preconditioning.

C. Remove validation monitoring.

D. Set all gradients to zero.

Correct Answer: B. Use momentum or improve conditioning through scaling or preconditioning.

Question 6

What limitation is associated with AdaGrad's accumulated squared-gradient denominator?

A. It cannot process sparse gradients.

B. Its effective learning rates may become extremely small over time.

C. It always increases the learning rate indefinitely.

D. It requires an exact Hessian.

Correct Answer: B. Its effective learning rates may become extremely small over time.

Question 7

Which update is characteristic of projected gradient descent?

A. The gradient step is followed by projection into the feasible set.

B. The objective is replaced by a random constraint.

C. All inequality multipliers are set to zero.

D. The Hessian is inverted at every iteration.

Correct Answer: A. The gradient step is followed by projection into the feasible set.

Question 8

In the illustrative credit-risk case, why was an unconstrained optimizer inadequate?

A. It could not calculate a logistic loss.

B. It minimized loss without necessarily satisfying governance constraints.

C. It always produces a convex solution.

D. It prevents feature standardization.

Correct Answer: B. It minimized loss without necessarily satisfying governance constraints.

Question 9

What does a nonzero gradient at a constrained optimum generally indicate?

A. The point cannot be optimal under any circumstances.

B. The gradient may be balanced by gradients of active constraints.

C. The objective must be constant everywhere.

D. The constraints have been removed.

Correct Answer: B. The gradient may be balanced by gradients of active constraints.

Question 10

Which observation most strongly suggests overfitting rather than merely poor optimization?

A. Training and validation losses both decrease together.

B. Training loss decreases while validation performance deteriorates.

C. The gradient norm is finite.

D. The model uses mini-batches.

Correct Answer: B. Training loss decreases while validation performance deteriorates.