Learning, Backpropagation and Optimization

Introduction

A neural network learns by adjusting numerical parameters so that its predictions become more accurate on a defined task. This process combines four ideas: forward propagation, which computes predictions; a loss function, which measures error; backpropagation, which computes how each parameter contributes to that error; and optimization, which updates parameters to reduce the loss.

These ideas form the computational core of supervised deep learning. They apply to multilayer perceptrons, convolutional networks, recurrent networks, transformers, and many other architectures. Although modern software performs most derivatives automatically, an understanding of the underlying mathematics is essential for diagnosing unstable training, selecting optimizers, interpreting learning curves, and designing reliable systems.

Learning Objectives

By the end of this chapter, you should be able to:

• derive forward propagation through a multilayer network;
• select suitable loss functions for regression and classification;
• explain gradient descent and mini-batch training;
• derive backpropagation using the chain rule;
• distinguish symbolic differentiation, numerical differentiation, and automatic differentiation;
• compare momentum, RMSProp, Adam, and related optimizers;
• use learning-rate schedules appropriately;
• explain the purpose of weight initialization and identify common training failures;
• apply these ideas to a practical machine-learning problem.

1. The Computational Model of a Neural Network

A neuron receives inputs, forms a weighted sum, adds a bias, and applies an activation function. For input vector x ∈ ℝⁿ, weight vector w ∈ ℝⁿ, bias b ∈ ℝ, and activation function φ, its output is

Here z is the pre-activation and a is the activation. The bias allows the neuron to shift its activation independently of the input. A layer containing m neurons is written

where W ∈ ℝᵐˣⁿ, b ∈ ℝᵐ, and φ is normally applied element by element. The dimensions must be compatible: an n-dimensional input produces an m-dimensional output.

Common activations include:

The derivative of ReLU is 1 for z > 0 and 0 for z < 0. At z = 0 it is not differentiable; implementations select a convenient subgradient.

2. Forward Propagation

Consider a network with L layers. Let a⁰ = x be the input. For layer l, where l = 1, …, L,

The final activation aᴸ is the prediction ŷ. Forward propagation evaluates these equations from the input layer to the output layer. During training, the intermediate values zˡ and aˡ are retained because backpropagation needs them.

For a batch of m examples, it is efficient to place examples in columns. If X ∈ ℝⁿˣᵐ, then

where 1 is an m-dimensional vector of ones. Broadcasting adds each bias vector to every example. Vectorization improves computational efficiency and produces consistent matrix dimensions.

Output-layer choices

For a real-valued regression target, the output may be linear: ŷ = zᴸ. For binary classification, a sigmoid output gives a probability p of the positive class. For K-class classification, softmax produces non-negative probabilities whose sum is 1. In practice, softmax and cross-entropy are often implemented together using a numerically stable log-sum-exp calculation rather than explicitly forming very large exponentials.

3. Loss Functions

A loss function ℓ(ŷ, y) measures error for one example, where y is the target and ŷ is the prediction. For a dataset of m examples, the empirical risk is commonly the mean loss:

The vector θ represents all trainable weights and biases. Regularization may be added:

where λ ≥ 0 controls the strength of the penalty and R is a regularization function.

Mean squared error

For regression, mean squared error is

It strongly penalizes large errors and is differentiable everywhere. If the target has extreme outliers, mean absolute error or a robust loss may be preferable.

Binary cross-entropy

For y ∈ {0, 1} and predicted probability p ∈ (0, 1), binary cross-entropy is

The logarithm is natural unless otherwise stated. The loss is low when the model assigns high probability to the correct class and becomes large when it is confidently wrong.

Multiclass cross-entropy

For one-hot target vector y and softmax probabilities p,

Because exactly one yₖ equals 1, this equals the negative logarithm of the probability assigned to the correct class. Cross-entropy is not the same as classification accuracy: accuracy uses a threshold or largest-probability decision, whereas cross-entropy evaluates the quality of the probabilities.

Regularization

L2 regularization adds

so its derivative with respect to θⱼ is θⱼ. L1 regularization uses R(θ)=∑_j|θ_j| and encourages sparsity, although it is not differentiable at zero. Regularization can reduce overfitting but does not replace validation data, good features, or appropriate model selection.

4. Gradient Descent

The gradient ∇θJ is the vector of partial derivatives of the objective with respect to all parameters. It points in the direction of greatest local increase. Therefore, basic gradient descent moves in the opposite direction:

Here η > 0 is the learning rate and t is the update index. The learning rate has no physical unit in ordinary machine-learning parameterizations, but it controls the scale of parameter change relative to the gradient.

There are three common data regimes:

For a mini-batch B of size |B|, the update uses

and θₜ₊₁ = θₜ − ηgₜ. Randomly shuffled mini-batches provide approximate gradients and can sometimes help the optimizer escape narrow regions or saddle points. However, the objective being optimized is still an empirical approximation to the desired population risk.

5. Backpropagation: Derivation and Interpretation

Backpropagation is an efficient application of the chain rule. It does not define a new kind of derivative; it organizes existing derivatives so that a network with many parameters can be differentiated with computational cost comparable to a small multiple of the forward pass.

For a single example, define the error signal at layer l as

For the output layer,

where ⊙ denotes elementwise multiplication. For an earlier layer,

The parameter gradients are

For a batch, gradients are summed or averaged across examples according to the loss convention. These equations assume column-vector activations and the layer definition zˡ = Wˡaˡ⁻¹ + bˡ.

Important special case: softmax with cross-entropy

When softmax is combined with multiclass cross-entropy, the output-layer gradient simplifies to

This simplification follows from differentiating both functions together. It is one reason the pairing is computationally and statistically useful.

Worked example

Consider a single linear neuron with x = 2, target y = 5, weight w = 1, and bias b = 0. The prediction is

ŷ = wx + b = 1×2 + 0 = 2.

Using the one-example squared loss ℓ = (1/2)(ŷ − y)²,

ℓ = (1/2)(2 − 5)² = 4.5.

Since ∂ℓ/∂ŷ = ŷ − y = −3, the parameter derivatives are

and ∂ℓ/∂b = ŷ − y = −3. With learning rate η = 0.1,

w_new = 1 − 0.1(−6) = 1.6,

b_new = 0 − 0.1(−3) = 0.3.

The new prediction is 1.6×2 + 0.3 = 3.5, which moves toward the target 5. This is one update, not a guarantee of immediate convergence.

6. Automatic Differentiation

Three methods are often confused:

• Symbolic differentiation manipulates algebraic expressions to produce a derivative expression.
• Numerical differentiation approximates a derivative using function evaluations, commonly (f(x+h)-f(x-h))/(2h). It is useful for checking gradients but can suffer from cancellation and step-size errors.
• Automatic differentiation decomposes a program into elementary operations and applies the chain rule exactly up to floating-point arithmetic.

Deep-learning libraries construct a computational graph during the forward pass. Reverse-mode automatic differentiation then traverses that graph backward, accumulating gradients. Reverse mode is efficient when a scalar loss depends on many parameters, which is the usual neural-network situation.

Automatic differentiation does not eliminate conceptual responsibility. Data leakage, incorrect target encoding, unsuitable losses, exploding gradients, and shape errors remain possible. Gradient checking on a small model can compare analytical or automatic gradients with finite-difference estimates, using a sufficiently small but not excessively small h.

7. Optimizers

Momentum

Momentum maintains a moving direction:

where 0 ≤ β < 1. Momentum smooths noisy gradients and accelerates progress along consistent directions. Some implementations use a factor (1 − β)gₜ; both conventions are valid if the learning rate is interpreted consistently.

RMSProp

RMSProp maintains an exponentially weighted average of squared gradients:

Operations on gₜ² and the square root are elementwise. ε > 0 prevents division by zero. Parameters with persistently large gradients receive smaller effective steps.

Adam

Adam combines momentum-like first moments and RMSProp-like second moments:

Because both moments start at zero, bias correction is used:

The update is

Typical starting values are β₁ = 0.9, β₂ = 0.999, and ε close to 10⁻⁸, but they are not universal laws. Adam often converges quickly, while SGD with momentum may sometimes produce better generalization after careful tuning. Optimizer choice should be evaluated on validation performance, not training loss alone.

8. Learning-Rate Schedules

A fixed learning rate may be adequate for a small problem, but schedules can improve training. Common approaches include:

• Step decay: reduce η by a factor after specified epochs.
• Exponential decay: η_t=η₀γ^t, where 0 < γ < 1.
• Cosine decay: gradually vary the rate according to a cosine curve.
• Warm-up: begin with a small rate and increase it during early updates.
• One-cycle schedules: increase and then decrease the rate over a planned training cycle.
• Reduce-on-plateau: lower the rate when validation performance stops improving.

A schedule should be linked to a measurable training objective. Excessively rapid decay can freeze learning; an excessive rate can cause divergence or oscillation. Learning-rate range tests, validation curves, and reproducible experiment logs are useful practical tools.

9. Weight Initialization and Signal Propagation

Poor initialization can make a network learn extremely slowly or not at all. Initializing all weights to zero is unsuitable for hidden layers because neurons receive identical gradients and remain identical. Biases can often be initialized to zero, although specialized architectures may use different choices.

For a layer with n_in inputs, Xavier/Glorot initialization is designed to maintain approximate activation variance for symmetric activations:

For ReLU-like activations, He initialization is commonly used:

A normal distribution with this variance or a corresponding uniform distribution may be used. These are variance heuristics, not guarantees. Their assumptions include roughly independent inputs, comparable feature scales, and a particular activation behavior. Batch normalization, residual connections, normalization layers, and careful preprocessing also affect signal propagation.

Feature scaling is important. If one input is measured in thousands and another in fractions, the geometry of the optimization problem may become poorly conditioned. Standardization commonly uses

where μ and σ are computed from the training set only. The same training-set statistics must be applied to validation and test data.

10. Diagnosing Training Problems

Gradient clipping limits the norm when it exceeds a chosen threshold c:

This changes the update direction only when necessary and is particularly common in recurrent or very deep models. It is a safeguard, not a substitute for finding the underlying cause.

11. Applied Case: Illustrative Credit-Default Classifier

Background and problem

A financial institution wants to estimate whether an applicant will default within twelve months. The available features include income, debt ratio, account history, and repayment indicators. This is an illustrative case, not a recommendation for real credit decisions.

Analysis

The target is binary, so a sigmoid output and binary cross-entropy are suitable. The data must be divided by time or by carefully designed sampling where appropriate; random splitting can produce leakage if multiple records from the same customer appear in different partitions. Numerical features are standardized using training-set statistics. Sensitive attributes should not be casually used as predictive inputs, and performance must be examined across relevant groups.

Approach

A small multilayer perceptron contains an input layer, two ReLU hidden layers, and one sigmoid output. Parameters are initialized with He initialization. Training uses mini-batches, Adam initially, early stopping based on validation loss, and a learning-rate reduction when validation loss plateaus. The model is evaluated using discrimination and calibration measures, not accuracy alone. For example, ROC-AUC assesses ranking, while calibration examines whether predicted probabilities correspond to observed frequencies.

Outcome and governance

Suppose validation cross-entropy decreases while the gap between training and validation loss grows. This suggests overfitting, so the team may add regularization, reduce model capacity, or collect more representative data. Even a numerically strong model should not be deployed without documenting data provenance, monitoring drift, investigating disparate error rates, providing appropriate explanations, and defining human review and appeal procedures. A prediction is not automatically a justified decision.

Learning points

The case demonstrates that optimization is inseparable from problem formulation. Correct gradients cannot repair biased labels, leakage, inappropriate metrics, or an unjustified deployment policy. Technical performance and responsible governance must be assessed together.

12. Best Practices

1. Begin with a simple baseline and a clearly defined metric.
2. Verify tensor shapes and gradients on a tiny synthetic dataset.
3. Separate training, validation, and test data before preprocessing.
4. Scale features using training data only.
5. Select the output activation and loss as a compatible pair.
6. Monitor both optimization loss and application-relevant validation metrics.
7. Log random seeds, data versions, hyperparameters, and software versions.
8. Tune the learning rate before making the network unnecessarily large.
9. Inspect gradient norms and activation distributions when training fails.
10. Reassess fairness, calibration, privacy, security, and drift before deployment.

Chapter Summary

Neural-network learning consists of a forward computation, a loss evaluation, gradient calculation, and parameter update. Forward propagation applies zˡ=Wˡaˡ⁻¹+bˡ and an activation at each layer. Loss functions translate prediction quality into an objective; mean squared error is common for regression, while cross-entropy is common for classification. Gradient descent updates parameters opposite the loss gradient, and mini-batches provide an efficient stochastic approximation.

Backpropagation applies the chain rule from the output toward the input. Automatic differentiation implements this process through a computational graph, while numerical differentiation is mainly a diagnostic tool. Momentum, RMSProp, and Adam modify gradient descent by incorporating historical gradient information. Learning-rate schedules regulate the scale of updates over time. Weight initialization, feature scaling, normalization, and activation choice strongly influence whether signals and gradients remain usable. Finally, successful optimization does not guarantee a valid AI system: data quality, evaluation design, fairness, privacy, monitoring, and human governance remain essential.

Key Terms

Forward propagation: Computing activations from inputs to outputs.

Pre-activation: The weighted sum plus bias before an activation function.

Activation function: A nonlinear function applied to a neuron’s pre-activation.

Logit: An unnormalized output score, especially before softmax or sigmoid.

Loss function: A scalar measure of prediction error.

Empirical risk: The average loss over a finite dataset.

Regularization: A method that penalizes model complexity.

Gradient: The vector of partial derivatives of an objective.

Gradient descent: Optimization by moving opposite the gradient.

Mini-batch: A subset of training examples used for one update.

Backpropagation: Efficient reverse application of the chain rule.

Error signal: The derivative of loss with respect to a layer’s pre-activation.

Automatic differentiation: Computational evaluation of exact chain-rule derivatives up to numerical precision.

Momentum: An optimizer mechanism that accumulates a moving direction.

RMSProp: An optimizer that scales updates using a moving average of squared gradients.

Adam: An adaptive optimizer combining first- and second-moment estimates.

Learning-rate schedule: A rule that changes the learning rate during training.

Weight initialization: The procedure for assigning starting parameter values.

Gradient clipping: Limiting gradient magnitude to improve numerical stability.

Calibration: Agreement between predicted probabilities and observed frequencies.

Review Questions

Short Answer Questions

1. What roles do forward propagation and the loss function play in supervised learning?
2. Why is the learning rate important in gradient descent?
3. State the backpropagation recurrence for a hidden layer.
4. How does automatic differentiation differ from numerical differentiation?
5. Why is initializing every hidden-layer weight to zero problematic?

Long Answer Questions

1. Derive the gradient for a single linear neuron using the half-squared-error loss.
2. Explain binary cross-entropy and why it is appropriate for sigmoid outputs.
3. Compare SGD with momentum, RMSProp, and Adam, including their update information.
4. Discuss how initialization, feature scaling, and activation functions affect gradient flow.
5. Explain how learning-rate schedules and validation monitoring can improve training reliability.

Analytical / Application Questions

1. A network’s training loss decreases but its validation loss increases after several epochs. Diagnose the likely problem and propose two technically justified responses.
2. Design an optimization plan for a multiclass image classifier, specifying output activation, loss, initialization, optimizer, schedule, and evaluation checks.
3. A credit-risk model has strong overall AUC but poor calibration and substantially different error rates across groups. Explain why deployment should be delayed and identify governance and technical analyses required before use.

Multiple Choice Questions with Answers

Question 1

Which expression represents a standard gradient-descent update?

A. θ_{t+1} = θ_t + η∇_θJ

B. θ_{t+1} = θ_t - η∇_θJ

C. θ_{t+1} = ηθ_t + ∇_θJ

D. θ_{t+1} = θ_t ÷ η

Correct Answer: B. θ_{t+1} = θ_t - η∇_θJ

Question 2

For multiclass classification with one-hot targets, which output and loss pairing is standard?

A. Linear output with mean absolute error

B. ReLU output with squared error

C. Softmax output with cross-entropy

D. Tanh output with hinge loss only

Correct Answer: C. Softmax output with cross-entropy

Question 3

What does backpropagation primarily compute?

A. The number of training examples

B. The model’s accuracy without predictions

C. Parameter gradients using the chain rule

D. A new activation function

Correct Answer: C. Parameter gradients using the chain rule

Question 4

For a softmax output combined with cross-entropy, the gradient with respect to logits is:

A. p + y

B. p − y

C. p × y

D. y ÷ p

Correct Answer: B. p − y

Question 5

What is the main purpose of bias-correction terms in Adam?

A. To remove all regularization

B. To correct the initialization of moment estimates at zero

C. To convert classification into regression

D. To guarantee zero training loss

Correct Answer: B. To correct the initialization of moment estimates at zero

Question 6

Which initialization is commonly associated with ReLU-like hidden units?

A. He initialization

B. All weights equal to zero

C. A distribution with variance 1/n_out only

D. One fixed positive constant

Correct Answer: A. He initialization

Question 7

If the learning rate is excessively large, which behavior is most plausible?

A. The loss may oscillate or diverge

B. All gradients become exactly zero by definition

C. The test set becomes larger

D. The model automatically becomes calibrated

Correct Answer: A. The loss may oscillate or diverge

Question 8

Which method uses a moving average of squared gradients to scale updates?

A. RMSProp

B. Plain batch gradient descent

C. One-hot encoding

D. Early stopping

Correct Answer: A. RMSProp

Question 9

For binary cross-entropy, if y = 1, which term remains in the loss?

A. −log(p)

B. −log(1−p) only

C. p²

D. 1−p²

Correct Answer: A. −log(p)

Question 10

Why should validation and test features use training-set scaling statistics?

A. To prevent information from evaluation data influencing preprocessing

B. To increase the number of labels

C. To guarantee perfect accuracy

D. To make every feature binary

Correct Answer: A. To prevent information from evaluation data influencing preprocessing