Optimization and Engineering Decision Models

Introduction

Optimization is the systematic search for the best feasible design, operating condition, estimate, or decision according to a stated objective. In mechatronics, optimization links mathematical modelling with practical engineering judgment. Typical problems include selecting actuator dimensions, tuning a controller, estimating uncertain parameters, allocating limited energy, and balancing cost against accuracy, speed, reliability, and safety.

An optimization model contains four essential components: decision variables, an objective function, constraints, and model parameters. A mathematically elegant solution is not automatically an engineering solution. The model must represent the physical system adequately, use consistent units, respect safety limits, and support a decision that is explainable and robust.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Formulate unconstrained and constrained engineering optimization problems.
2. Use gradients, Hessians, and second-order conditions to classify stationary points.
3. Apply Lagrange multipliers to equality-constrained problems.
4. Explain and apply numerical methods such as steepest descent, Newton’s method, and penalty methods.
5. Formulate parameter-estimation, resource-allocation, and control-tuning models.
6. Interpret multi-criteria decisions, sensitivity, feasibility, and trade-offs.
7. Evaluate optimization results critically with respect to uncertainty, ethics, and implementation.

1. Mathematical Structure of an Optimization Problem

A general minimization problem is written as

Here, x = [x₁, x₂, …, xₙ]ᵀ is the vector of decision variables; f(x) is the objective function; gᵢ(x) are inequality constraints; hⱼ(x) are equality constraints; and X specifies domains such as xᵢ ≥ 0 or integer restrictions. A maximization problem can be converted to minimization by minimizing −f(x).

A point is feasible if it satisfies every constraint. The feasible region is the set of all feasible points. A global optimum is at least as good as every feasible point; a local optimum is better than nearby feasible points. Numerical algorithms commonly find local optima, so convexity and multiple starting points are important.

Variables, parameters, and units

Decision variables are quantities selected by the designer, such as motor torque or sampling time. Parameters are known or estimated quantities, such as mass, resistance, or a measured reference. The objective and constraints must be dimensionally meaningful. If cost in dollars is added to energy in joules, weighting or normalization is required; direct addition is invalid.

A useful engineering model separates performance from restrictions:

2. Unconstrained Optimization

For a differentiable scalar function f(x), an interior local optimum x* must satisfy the first-order condition

The gradient ∇f is the column vector of first partial derivatives. In two variables,

The condition is necessary for an interior optimum, not sufficient. A stationary point may be a minimum, maximum, or saddle point.

The Hessian matrix is

At a stationary point, if H(x) is positive definite, x is a strict local minimum. If it is negative definite, x is a strict local maximum. If it is indefinite, x is a saddle point. For two variables, positive definiteness can be checked using

where D is the Hessian determinant. If D < 0, the point is a saddle point. These tests assume twice-continuous differentiability near the point.

Worked Example 1: Design trade-off

Suppose a simplified actuator design has objective

where x and y are design parameters in their respective engineering units, and J is a normalized cost. Differentiating gives

Setting the gradient to zero gives x = 3 and y = 1. The Hessian is

Both eigenvalues are positive, so H is positive definite. Therefore, (3, 1) is the unique global minimum for this quadratic function, with J = 4. The larger curvature in the y direction means that deviations in y are more costly than equally sized deviations in x.

3. Numerical Optimization Methods

Analytical solutions are uncommon for nonlinear engineering models. Numerical optimization generates a sequence x⁽ᵏ⁾ approaching a solution.

Steepest descent

The steepest-descent iteration is

where αₖ > 0 is the step length. The negative gradient is the direction of greatest local decrease. A fixed step may be too slow or may diverge; line search selects αₖ to reduce the objective while preserving stability.

For f(x) = ½xᵀQx − bᵀx, where Q is symmetric positive definite, the function is strictly convex and has a unique minimum. Poor conditioning of Q causes narrow valleys and slow zig-zagging. Scaling variables and using preconditioning can improve performance.

Newton’s method

Newton’s method uses curvature:

when the Hessian is nonsingular. Near a sufficiently regular solution, convergence can be very fast. However, computing and inverting a Hessian may be expensive, and an indefinite Hessian can produce a direction that is not descending. Practical implementations use damping, trust regions, or quasi-Newton approximations such as BFGS.

A numerical result should be checked using objective change, variable change, gradient norm, constraint violation, and physical plausibility. Termination based only on iteration count is unsafe.

4. Equality-Constrained Optimization and Lagrange Multipliers

Consider minimizing f(x) subject to h(x) = 0. The Lagrangian is

where λ is a Lagrange multiplier. Necessary first-order conditions are

For several constraints, λ is a vector and the term becomes ∑ λⱼhⱼ(x). Geometrically, at a constrained optimum the objective gradient is a linear combination of the active constraint gradients. Thus, the objective cannot decrease while remaining on the constraint surface.

Under regularity conditions, the multiplier also indicates sensitivity. If the constraint is written h(x) = b, the multiplier is related to the change in optimal objective value as b changes. Its exact sign depends on the chosen Lagrangian convention. Consequently, units of λ are objective units divided by constraint units.

Worked Example 2: Minimum-energy allocation

Two actuators receive nonnegative inputs u₁ and u₂. The required total input is 10 units, and energy cost is

Minimize E subject to u₁ + u₂ = 10. The Lagrangian is

Stationarity gives 2u₁ + λ = 0 and 4u₂ + λ = 0. Therefore u₁ = 2u₂. Combining this with u₁ + u₂ = 10 gives 3u₂ = 10, so

The minimum energy is

The cheaper actuator receives more input, but both actuators operate because the equality constraint requires their combined contribution.

5. Inequality Constraints and KKT Conditions

For inequalities, the relevant framework is the Karush–Kuhn–Tucker (KKT) conditions. For minimization with gᵢ(x) ≤ 0, define

The conditions are:

1. Stationarity: ∇ₓL = 0.
2. Primal feasibility: hⱼ(x) = 0 and gᵢ(x) ≤ 0.
3. Dual feasibility: μᵢ ≥ 0.
4. Complementary slackness: μᵢgᵢ(x) = 0.

Complementary slackness means either a constraint is inactive, gᵢ(x) < 0 and μᵢ = 0, or active, gᵢ(x) = 0 and μᵢ may be positive. For convex problems with convex f and gᵢ and affine equality constraints, KKT conditions are also sufficient under standard constraint qualifications. For nonconvex problems they are generally only necessary, and a KKT point may not be globally optimal.

6. Penalty and Barrier Approaches

A constrained problem can be approximated by an unconstrained one. A quadratic penalty function is

where r > 0 is a penalty parameter. Constraint violations become expensive as r increases. Very large r can make the problem ill-conditioned, so penalty methods may require a sequence of moderate values.

An interior barrier for inequalities is commonly written

with μ > 0 and gᵢ(x) < 0. The logarithm requires strict feasibility and prevents iterates from crossing the boundary. As μ decreases, solutions approach the constrained optimum under suitable assumptions. Penalty and barrier methods are useful concepts, but professional software often uses sequential quadratic programming, interior-point algorithms, or trust-region methods.

7. Engineering Formulations

Parameter estimation

Suppose measured outputs yₖ are predicted by a model φ(tₖ,θ), where θ is an unknown parameter vector. Least squares estimates θ by minimizing

For linear regression y = Xθ + ε, the ordinary least-squares estimate is

when XᵀX is nonsingular. This assumes an appropriate model, independent errors with zero mean, and— for the usual statistical efficiency interpretation—constant error variance. Poorly conditioned or nearly dependent columns in X make estimates sensitive to noise. Regularization can be introduced as

where ρ ≥ 0 controls the trade-off between data fit and parameter magnitude.

Resource allocation

A production or energy-allocation model may maximize benefit:

where cᵢ is benefit per unit, aᵢ resource consumption, b available resource, and xᵢ allocation. Linear programming is appropriate when relationships are linear and variables are continuous. Integer or mixed-integer models are needed for choices such as the number of robots or whether a machine is switched on.

Control tuning

A controller can be tuned by minimizing a performance index such as

where K is the gain vector, e is tracking error, u is control effort, T is the evaluation horizon, and ρ ≥ 0 weights effort relative to error. Constraints may enforce actuator saturation, settling time, overshoot, stability margins, or thermal limits. Simulation-based optimization must distinguish numerical stability in simulation from guaranteed closed-loop stability.

8. Multi-Criteria Decision Models

Engineering decisions rarely have one objective. Cost, mass, precision, speed, reliability, and environmental impact may conflict. A weighted-sum model uses normalized criteria:

where zᵢ are dimensionless normalized scores and wᵢ are stated preferences. For a benefit criterion, one possible normalization is

For a cost criterion, reverse the numerator. Normalization must reflect meaningful bounds; arbitrary scaling can change the decision.

A Pareto-optimal solution is one for which no objective can improve without worsening at least one other objective. The set of such solutions is the Pareto front. Weighted sums may miss nonconvex portions of the front, so ε-constraint methods, goal programming, or direct Pareto search may be preferable.

Decision matrices and analytic hierarchy processes can organize qualitative judgments, but rankings depend on criteria definitions, weights, and uncertainty. Sensitivity analysis should vary important weights and parameters rather than presenting one ranking as objectively inevitable.

9. Illustrative Case: Energy-Efficient Pick-and-Place Robot

Background

Consider a two-axis pick-and-place robot in a manufacturing cell. The design team must select a motion profile and controller gains. Faster motion improves throughput, while high acceleration increases motor current, vibration, and energy use.

Problem formulation

Let x₁ be cycle time in seconds and x₂ be normalized control effort. An illustrative objective is

The coefficients are dimensionless preference weights after normalization. Suppose testing produces a feasible relation x₂ = 1/x₁ for x₁ > 0, and safety requires x₁ ≥ 1. The problem becomes

Differentiation gives

The stationary point satisfies x₁² = 0.40/0.60 = 2/3, so x₁ ≈ 0.816. This violates x₁ ≥ 1. Since J′(x₁) > 0 for x₁ ≥ 1, the constrained optimum is the boundary x₁ = 1, with x₂ = 1 and J = 1.00.

Interpretation

The unconstrained mathematical preference would be faster than the safe operating region, but the constraint changes the design. The result should not be accepted without validating the simplified relation, checking actuator temperature, testing repeatability, and considering wear and operator safety. If the weights change, the preferred point may change; this is a decision model, not a physical law.

Outcome and learning points

The case demonstrates four principles: formulate the objective and constraints separately; test whether an unconstrained solution is feasible; expect active constraints in engineering design; and communicate the effect of preferences and assumptions. A practical implementation would perform experiments or high-fidelity simulation, estimate uncertainty, and re-optimize when payload or supply voltage changes.

10. Verification, Robustness, and Professional Practice

Optimization outputs require verification. First, independently substitute the solution into every constraint. Second, compare against simple benchmark designs. Third, perturb parameters and initial guesses. A robust design remains acceptable under plausible uncertainty, even if it is not optimal for nominal values.

Robust optimization may impose performance for all parameter values in an uncertainty set. Stochastic optimization instead represents uncertain quantities probabilistically and may minimize expected cost or impose a probability-of-failure limit. These approaches are more realistic but require credible uncertainty models.

Common failure modes include optimizing an incomplete objective, using incompatible units, ignoring infeasibility, overfitting measurement noise, accepting a local optimum without investigation, and reporting excessive numerical precision. Engineers should document data sources, assumptions, software versions, tolerances, constraints, and validation tests.

Safety-critical decisions require independent review and conservative margins. An optimization algorithm must not override legal requirements, machine safeguarding, cybersecurity controls, or professional responsibility. AI-assisted optimization can explore designs rapidly, but generated models and recommendations require human verification, traceability, bias assessment, and protection of proprietary or safety-sensitive data.

Chapter Summary

Optimization converts engineering choices into explicit mathematical models containing variables, objectives, constraints, and parameters. Gradients identify first-order stationary points, while Hessians classify unconstrained solutions under differentiability and definiteness conditions. Steepest descent is simple but may be slow; Newton and quasi-Newton methods use curvature and require careful numerical safeguards. Lagrange multipliers handle equality constraints, and KKT conditions extend the analysis to inequalities through feasibility, stationarity, dual feasibility, and complementary slackness.

Engineering applications include least-squares parameter estimation, resource allocation, controller tuning, and multi-criteria design. Normalization and weighting make competing objectives comparable, while Pareto analysis reveals trade-offs. A credible optimization result must be feasible, dimensionally consistent, numerically verified, physically validated, and tested for sensitivity and robustness. Optimization supports engineering judgment; it does not replace it.

Key Terms

Decision variable: Quantity selected by the designer or decision-maker.

Objective function: Mathematical measure to minimize or maximize.

Constraint: Condition that feasible solutions must satisfy.

Feasible region: Set of points satisfying all constraints.

Local optimum: Best solution within a neighborhood.

Global optimum: Best solution over the entire feasible region.

Gradient: Vector of first partial derivatives.

Hessian: Matrix of second partial derivatives.

Convex function: Function whose line segment lies above its graph in the appropriate inequality sense.

Lagrangian: Objective augmented by weighted constraints.

Lagrange multiplier: Variable associated with an equality constraint and its sensitivity.

KKT conditions: First-order conditions for constrained optimization.

Complementary slackness: Condition linking inequality activity and its multiplier.

Penalty method: Method that incorporates constraint violations into an augmented objective.

Least squares: Estimation method minimizing squared residuals.

Regularization: Addition of a term that discourages undesirable parameter values.

Pareto optimum: Solution not improvable in one criterion without harming another.

Sensitivity analysis: Study of solution changes caused by parameter or preference changes.

Robust optimization: Optimization accounting explicitly for uncertainty.

Review Questions

Short Answer Questions

1. Distinguish a decision variable from a model parameter.
2. Why does a zero gradient not guarantee a local minimum?
3. State the four categories of KKT conditions.
4. What is the purpose of normalization in a weighted multi-criteria model?
5. Why should an optimization result be tested against parameter perturbations?

Long Answer Questions

1. Explain the role of the Hessian in classifying stationary points, including the assumptions required.
2. Derive the Lagrange multiplier equations for minimizing a differentiable function subject to two equality constraints.
3. Compare steepest descent, Newton’s method, and quasi-Newton methods in terms of information required, convergence, and implementation risks.
4. Explain how least-squares parameter estimation can be affected by measurement noise, model error, and ill-conditioned data.
5. Discuss why a Pareto front may be more informative than a single weighted-sum solution in mechatronic design.

Analytical / Application Questions

1. Minimize f(x,y) = (x − 2)² + 3(y − 4)² subject to x + y = 5 using Lagrange multipliers. Verify feasibility and classify the solution.
2. A controller designer must trade tracking error against energy. Propose a dimensionally consistent objective, identify at least three constraints, and explain how the weights could be selected and tested.
3. Formulate a resource-allocation model for two actuators sharing a limited electrical-energy budget. Include nonnegativity, a performance requirement, and one uncertainty or robustness consideration.

Multiple Choice Questions with Answers

Question 1

Which condition is necessary at an interior differentiable unconstrained local optimum?

A. The gradient is zero

B. Every variable is zero

C. The Hessian is zero

D. All constraints are active

Correct Answer: A. The gradient is zero

Question 2

A positive-definite Hessian at a stationary point indicates a strict local:

A. Maximum

B. Minimum

C. Saddle point

D. Infeasible point

Correct Answer: B. Minimum

Question 3

In the KKT conditions, complementary slackness is written as:

A. μᵢ + gᵢ(x) = 0

B. μᵢgᵢ(x) = 0

C. μᵢgᵢ(x) = 1

D. μᵢ ≤ gᵢ(x)

Correct Answer: B. μᵢgᵢ(x) = 0

Question 4

For a constraint written gᵢ(x) ≤ 0, dual feasibility requires:

A. μᵢ ≥ 0

B. μᵢ ≤ 0

C. gᵢ(x) ≥ 1

D. μᵢ = 1

Correct Answer: A. μᵢ ≥ 0

Question 5

The steepest-descent update uses which direction?

A. The positive gradient

B. The negative gradient

C. The Hessian determinant

D. The constraint value

Correct Answer: B. The negative gradient

Question 6

For E = u₁² + 2u₂² with u₁ + u₂ = 10, the minimum-energy allocation is:

A. u₁ = 5, u₂ = 5

B. u₁ = 10, u₂ = 0

C. u₁ ≈ 6.667, u₂ ≈ 3.333

D. u₁ ≈ 3.333, u₂ ≈ 6.667

Correct Answer: C. u₁ ≈ 6.667, u₂ ≈ 3.333

Question 7

In ordinary least squares, θ̂ = (XᵀX)⁻¹Xᵀy requires, among other conditions, that:

A. XᵀX is nonsingular

B. All measurements are zero

C. The model has no parameters

D. The residuals are always exactly zero

Correct Answer: A. XᵀX is nonsingular

Question 8

A Pareto-optimal design has the property that:

A. Every objective is individually minimized

B. No criterion can improve without worsening at least one other criterion

C. All weights must be equal

D. All constraints are inactive

Correct Answer: B. No criterion can improve without worsening at least one other criterion

Question 9

The logarithmic barrier −μ∑ln(−gᵢ(x)) requires:

A. gᵢ(x) > 0

B. gᵢ(x) = 1

C. gᵢ(x) < 0

D. μ = 0

Correct Answer: C. gᵢ(x) < 0

Question 10

In the illustrative robot case, the constrained optimum occurs at x₁ = 1 because the unconstrained stationary point:

A. Has an infinite objective

B. Violates the safety constraint x₁ ≥ 1

C. Makes x₂ negative

D. Is a maximum

Correct Answer: B. Violates the safety constraint x₁ ≥ 1