Neural Network Learning and Backpropagation

Introduction

A feedforward neural network is a parameterized function that transforms an input vector into an output through a sequence of layers. Its parameters—weights and biases—are learned from data by minimizing a loss function. This chapter develops the mathematical and engineering foundations of that process: network construction, activation and loss selection, parameter initialization, forward propagation, backpropagation, and gradient-based optimization.

The central idea is simple but powerful. A network first computes a prediction, measures the discrepancy between that prediction and the target, and then propagates information about the discrepancy backward through the computational graph. The resulting gradients indicate how each parameter should change to reduce the loss.

Learning Objectives

By the end of this chapter, you should be able to:

1. Construct and interpret multilayer feedforward networks.
2. Select activation functions and output-layer transformations for different tasks.
3. Choose suitable loss functions and explain their probabilistic interpretation.
4. Explain the purpose of parameter initialization and identify common failure modes.
5. Derive forward propagation and backpropagation using the chain rule.
6. Apply gradient descent and its variants to train neural networks.
7. Diagnose vanishing gradients, exploding gradients, overfitting, and unstable optimization.
8. Evaluate design choices in a practical deep-learning application.

1. Feedforward Networks as Compositions of Functions

A feedforward network contains directed connections from an input layer through one or more hidden layers to an output layer. Information moves in one direction; there are no recurrent connections or cycles.

Let the input be x ∈ ℝ^n₀. For layer l, define:

Here, a^(0) = x; W^(l) is the weight matrix; b^(l) is the bias vector; z^(l) is the pre-activation; φ^(l) is the activation function; and a^(l) is the layer output. If layer l−1 has n_l−1 units and layer l has n_l units, then W^(l) ∈ ℝ^{n_l × n_l−1} and b^(l) ∈ ℝ^n_l.

The complete network represents the composition:

where θ denotes all weights and biases. The term feedforward describes the information flow, not the learning procedure: training still requires repeated forward and backward computations.

Width, depth, and representational capacity

The width of a network is the number of units in a layer; depth is the number of computational layers. Increasing depth allows the network to represent hierarchical transformations, while increasing width provides more units for representing functions at a given level. A linear network, regardless of depth, is equivalent to a single linear transformation because a composition of linear functions remains linear. Nonlinear activations are therefore essential for expressive multilayer models.

The universal approximation theorem states, under suitable conditions, that a sufficiently wide network with a nonconstant nonlinear activation can approximate broad classes of continuous functions on compact domains. This is an existence result, not a guarantee that training will find the required parameters or that the approximation will generalize well.

2. Activation Functions

An activation function introduces nonlinearity and determines how signals and gradients pass through a layer.

Sigmoid

Its output lies in (0, 1), and its derivative is:

The sigmoid is useful for a binary-probability output, but it can saturate when z is very positive or negative. In saturated regions, the derivative approaches zero, producing weak gradients. It is therefore rarely used in modern hidden layers.

Hyperbolic tangent

Its range is (−1, 1), and tanh′(z) = 1 − tanh²(z). It is zero-centred but still saturates for large absolute inputs.

ReLU and related functions

Its derivative is 1 for z > 0 and 0 for z < 0; at zero, a subgradient convention is chosen in software. ReLU is computationally inexpensive and generally provides stronger gradients than sigmoid or tanh in its active region. A unit can, however, become permanently inactive if its inputs remain negative.

Leaky ReLU addresses this risk:

where α is commonly small. Smooth alternatives include GELU and SiLU, which are frequently used in modern architectures. Their practical advantage depends on architecture, initialization, normalization, and optimization rather than on the activation alone.

Softmax

For a K-class output, softmax converts logits into a probability vector:

Each output is positive and the outputs sum to one. Numerically stable implementations subtract max_j z_j before exponentiation; this does not change the result because adding the same constant to every logit leaves softmax unchanged.

3. Output Layers and Loss Functions

The output transformation and loss function should match the prediction task and the assumed data-generating process.

Mean squared error

For one scalar target y and prediction ŷ,

The factor one-half is conventional because it simplifies differentiation. For m examples, the empirical mean is:

MSE strongly penalizes large errors and is appropriate when errors are plausibly Gaussian with constant variance. It can be sensitive to outliers.

Binary cross-entropy

For target y ∈ {0,1} and predicted probability p,

The logarithm is natural unless otherwise specified. Predictions should be clipped only for numerical protection; in practice, stable implementations combine logits and the loss directly.

Multiclass cross-entropy

For one-hot target vector y and softmax prediction p,

If exactly one class c is correct, this reduces to L_CE = −log(p_c). Cross-entropy is the negative log-likelihood of a categorical model. It therefore rewards assigning high probability to the correct class and heavily penalizes confident incorrect predictions.

Loss, risk, and regularization

The empirical objective is an average over the training set. Generalization concerns performance on unseen data. Regularization adds a preference for certain parameter values. L2 regularization, also called weight decay in some optimization settings, adds:

where λ ≥ 0 controls the penalty and ||·||_F is the Frobenius norm. Biases are often excluded. The gradient contribution for a weight matrix is λW^(l). Regularization changes the optimization objective; it is not a substitute for validation data, sound feature design, or appropriate model capacity.

4. Initialization and Data Scaling

Initialization is not a minor implementation detail. If all hidden units begin with identical parameters, they receive identical gradients and remain identical, a phenomenon called symmetry failure. Random initialization breaks this symmetry.

Very small weights can cause signals and gradients to shrink across layers. Very large weights can cause activations and gradients to grow or saturate. Variance-preserving schemes address this problem approximately under assumptions about independent inputs and activations.

For a layer with n_in inputs, Xavier or Glorot initialization is commonly used for tanh-like activations. A uniform version is:

For ReLU-like activations, He initialization is often preferred:

The notation Normal(0, 2/n_in) means mean zero and variance 2/n_in, not standard deviation 2/n_in. Biases are commonly initialized to zero, although specialized architectures may use other choices.

Inputs should generally be standardized using training-set statistics:

where μ_j and σ_j are computed on the training data only. Applying statistics from the validation or test set causes data leakage.

5. Forward Propagation: A Worked Example

Consider a network with two inputs, two hidden units, and one sigmoid output. Let

x = [1, 2]^T,

and

Use ReLU in the hidden layer and sigmoid at the output.

First,

Therefore,

The output pre-activation is

Thus,

The network predicts a probability of approximately 0.321 for the positive class. If the target is y = 1, the binary cross-entropy is approximately −log(0.3208) ≈ 1.137.

6. Backpropagation and the Chain Rule

Backpropagation is an efficient application of the chain rule to a layered computational graph. It computes exact derivatives of the loss with respect to all parameters for a fixed forward pass. It is not itself an optimization algorithm; gradient descent uses the derivatives produced by backpropagation.

For a single example, define the output error signal:

For a general activation at the output,

where ⊙ denotes elementwise multiplication. The parameter gradients are:

For a batch, these quantities are summed or averaged consistently with the chosen loss reduction.

Important simplification: sigmoid with binary cross-entropy

Let ŷ = σ(z) and use BCE. Differentiation gives:

This cancellation avoids separately multiplying the sigmoid derivative in the final expression. Similarly, softmax combined with categorical cross-entropy yields:

These identities assume the stated activation-loss pair and the standard unweighted loss. Class weighting or label smoothing changes the details.

Continuing the worked example

Using ŷ ≈ 0.3208 and y = 1,

The output gradients are

and ∂L/∂b^(2) ≈ −0.6792. For the hidden layer,

because the first hidden pre-activation was negative and the second was positive. Therefore,

and ∂L/∂b^(1) = [0, 0.3396]^T. These values specify the direction of steepest local increase in loss; training moves parameters in the opposite direction.

7. Gradient-Based Learning

For learning rate η > 0, batch gradient descent updates parameters by:

Stochastic gradient descent uses one example at a time; mini-batch SGD uses a subset. Mini-batches provide computational efficiency and a useful amount of gradient noise, which can sometimes improve generalization.

A practical training loop is:

1. Sample a mini-batch.
2. Perform the forward pass.
3. Compute the loss.
4. Run backpropagation.
5. Apply an optimizer update.
6. Monitor training and validation metrics.
7. Stop or adjust the schedule according to validation behavior.

Momentum and adaptive methods

Momentum maintains a velocity:

where g_t = ∇_θJ(θ_t) and 0 ≤ β < 1. Adam maintains first- and second-moment estimates of gradients and applies bias correction. It often converges quickly, but SGD with momentum may sometimes provide better final generalization. No optimizer removes the need for learning-rate selection, validation, and diagnosis.

Gradient clipping limits unusually large updates. For global norm clipping with threshold c, the gradient is replaced by:

This is especially useful when exploding gradients destabilize training, although persistent explosion usually signals a deeper issue with architecture, initialization, data, or learning rate.

8. Diagnosing Training Behaviour

Vanishing gradients occur when repeated derivatives have magnitudes substantially below one, causing early layers to learn slowly. Saturating sigmoid or tanh units and excessively deep compositions are common causes. Exploding gradients occur when repeated Jacobian factors amplify signals. ReLU-family activations, suitable initialization, normalization, residual connections, and carefully chosen learning rates can reduce these risks.

Underfitting occurs when both training and validation performance are poor; possible responses include increasing capacity, improving features, reducing excessive regularization, or training longer. Overfitting occurs when training performance improves while validation performance deteriorates. Responses include more data, augmentation, regularization, early stopping, reduced capacity, or improved validation methodology.

A useful diagnostic practice is to inspect:

• training and validation loss curves;
• gradient norms by layer;
• activation distributions and dead ReLU rates;
• calibration and class-specific metrics;
• sensitivity to random seeds;
• performance under distribution shift.

Accuracy alone can be misleading for imbalanced classification. Precision, recall, F1 score, area under suitable ranking curves, calibration, and decision-specific costs should be considered.

9. Case Study: Predictive Maintenance for Industrial Pumps

Background

This illustrative case concerns a manufacturer operating centrifugal pumps. Each pump produces temperature, vibration, pressure, and motor-current measurements. The objective is to predict whether a failure will occur within the next 24 hours.

Problem

Failures are rare, labels are time-dependent, and false negatives are expensive because they can cause production stoppage. Randomly splitting individual sensor rows would leak information across time and possibly place measurements from the same failure episode in both training and test sets.

Analysis

Engineers first create windows of recent measurements and assign a label based only on future events. The data is split chronologically by operating period or by pump, depending on the deployment scenario. Features are standardized using training data only. Because the positive class is rare, the team evaluates recall, precision-recall behaviour, calibration, and the economic cost of alarms rather than accuracy alone.

Approach and solution

A feedforward baseline receives engineered window statistics—means, standard deviations, slopes, and recent values. It contains two hidden layers with ReLU activations, He initialization, and a sigmoid output. BCE is used with carefully selected class weighting. Dropout and L2 regularization are compared using a validation period. Early stopping is based on validation loss and operational recall.

The final threshold is not automatically set to 0.5. Instead, it is selected using the cost of inspection, downtime, and missed failures. Probability calibration is checked on a temporally separate set. A monitoring system tracks sensor drift, missingness, alarm rates, and post-deployment precision.

Outcome

The baseline reduces missed failures relative to a rules-based system, but the most important result is procedural: leakage-resistant evaluation and threshold selection produce a more credible estimate of operational value. The model is deployed initially as decision support, with engineers retaining authority to schedule maintenance.

Learning points

1. Architecture selection cannot compensate for flawed labels or leakage.
2. The loss function and decision threshold address different problems: the loss trains the model, while the threshold supports an operational decision.
3. Rare-event applications require metrics and validation designs aligned with deployment.
4. Monitoring and human oversight are part of the learning system, not postscript activities.

10. Implementation Best Practices and Governance

Use numerically stable library losses that accept logits where available. Verify tensor shapes and reductions explicitly; an accidental sum instead of mean can change the effective learning rate with batch size. Perform gradient checking on a small network using finite differences:

where ε is small but not so small that floating-point cancellation dominates. Gradient checking is a debugging tool, not a replacement for analytical backpropagation.

Reproducibility requires recording data versions, preprocessing, random seeds, architecture, optimizer, learning-rate schedule, stopping rule, and evaluation splits. In high-stakes settings, document intended use, known limitations, subgroup performance, uncertainty, access controls, and escalation procedures. Training data may contain personal or sensitive information; data minimization, lawful processing, retention controls, and security are essential. Models should not silently convert probabilistic outputs into consequential decisions without appropriate human review and accountability.

Contemporary systems increasingly use automatic differentiation, normalization, residual connections, and specialized accelerators. These tools automate derivative calculation and improve engineering productivity, but they do not alter the underlying chain rule. The researcher or engineer remains responsible for verifying objectives, gradients, data provenance, and generalization claims.

Chapter Summary

A feedforward neural network is a composition of affine transformations and nonlinear activations. Weights and biases determine the transformation, while activation functions determine both representational capacity and gradient flow. Output activations and losses must match the prediction task: sigmoid with binary cross-entropy for binary outcomes, softmax with categorical cross-entropy for mutually exclusive classes, and suitable regression losses for continuous targets.

Initialization breaks symmetry and helps preserve useful signal scales. Xavier initialization is commonly associated with tanh-like activations, while He initialization is often suitable for ReLU-like layers. Forward propagation computes predictions. Backpropagation applies the chain rule efficiently to obtain parameter gradients, and gradient-based optimizers use those gradients to update parameters. Stable training additionally requires appropriate scaling, learning rates, regularization, validation, diagnostics, and leakage-resistant evaluation.

In professional applications, model quality depends on more than architecture. Labels, data splitting, class imbalance, threshold selection, calibration, monitoring, privacy, and human governance determine whether a trained network is reliable and responsible in practice.

Key Terms

Activation function: Nonlinear function applied to a layer’s pre-activation.

Backpropagation: Efficient chain-rule procedure for computing gradients in a computational graph.

Bias: Trainable additive parameter in a neuron or layer.

Binary cross-entropy: Negative log-likelihood loss for a Bernoulli target.

Computational graph: Representation of operations and dependencies used for differentiation.

Cross-entropy: Loss measuring the negative log-likelihood of a target distribution under predicted probabilities.

Feedforward network: Network in which information flows from input to output without cycles.

Gradient: Vector of partial derivatives of a scalar objective with respect to parameters.

He initialization: Variance-scaled initialization commonly used with ReLU-like activations.

Logit: Pre-activation value supplied to a sigmoid or softmax transformation.

Loss function: Scalar measure of prediction error for an example or batch.

Mini-batch: Subset of training examples used to estimate a gradient.

Momentum: Optimization mechanism that smooths updates using a moving direction estimate.

Overfitting: Strong training performance combined with inadequate generalization.

Parameter: Learnable quantity, such as a weight or bias.

ReLU: Activation defined by max(0,z).

Softmax: Transformation of logits into nonnegative values summing to one.

Vanishing gradient: Gradient becoming too small for effective learning.

Weight decay: Regularization or optimizer mechanism that discourages large weights.

Review Questions

5 Short Answer Questions

1. Why are nonlinear activation functions necessary in multilayer feedforward networks?
2. State the dimensions of W^(l) for a layer with n_l−1 inputs and n_l units.
3. What is the purpose of random initialization?
4. Distinguish forward propagation from backpropagation.
5. Why should standardization statistics be computed using training data only?

5 Long Answer Questions

1. Explain sigmoid, tanh, ReLU, and softmax, including their ranges, uses, and gradient-related limitations.
2. Derive the backpropagation recurrence for a hidden layer using the chain rule.
3. Compare MSE, binary cross-entropy, and categorical cross-entropy in terms of task suitability and interpretation.
4. Explain how initialization, activation choice, normalization, and learning rate interact during training.
5. Discuss how overfitting and underfitting can be diagnosed and addressed in a neural-network project.

3 Analytical / Application Questions

1. Design a feedforward network for predicting loan default from tabular data. Specify the output activation, loss, initialization approach, evaluation metrics, and two governance safeguards.
2. A network’s training loss decreases rapidly, while validation loss begins increasing after several epochs. Analyze at least three possible causes or responses and explain how you would test them.
3. In the worked network, replace the target y = 1 with y = 0. Recompute the output error signal for sigmoid plus BCE and explain how the direction of the output-layer update changes.

Multiple Choice Questions with Answers

Question 1

Which component introduces nonlinear representational capacity into a multilayer feedforward network?

A. Bias initialization

B. Activation function

C. Mini-batch size

D. Feature scaling

Correct Answer: B. Activation function

Question 2

For binary classification with a sigmoid output, which loss is typically appropriate?

A. Binary cross-entropy

B. Categorical accuracy only

C. Frobenius norm of the input

D. Unweighted cosine distance

Correct Answer: A. Binary cross-entropy

Question 3

For sigmoid activation combined with binary cross-entropy, what is the derivative of the loss with respect to the output logit?

A. y − ŷ

B. ŷ − y

C. ŷ(1 − ŷ)

D. −log(ŷ)

Correct Answer: B. ŷ − y

Question 4

What is the principal purpose of random weight initialization in hidden layers?

A. To guarantee zero training loss

B. To eliminate the need for validation

C. To break symmetry between units

D. To make all activations identical

Correct Answer: C. To break symmetry between units

Question 5

Which initialization is commonly associated with ReLU-like activations?

A. He initialization

B. Zero variance initialization

C. Uniform initialization with variance independent of fan-in

D. One-valued initialization for every weight

Correct Answer: A. He initialization

Question 6

What does the learning-rate parameter control in gradient descent?

A. The number of output classes

B. The size of each parameter update

C. The number of hidden units

D. The validation-set size

Correct Answer: B. The size of each parameter update

Question 7

For a mutually exclusive K-class problem, what property does softmax output satisfy?

A. Every output is negative

B. Outputs are independent and need not sum to one

C. Outputs are nonnegative and sum to one

D. Exactly two outputs are nonzero

Correct Answer: C. Outputs are nonnegative and sum to one

Question 8

Which situation is characteristic of overfitting?

A. Both training and validation performance remain poor

B. Training performance improves while validation performance worsens

C. Training and validation losses are exactly constant

D. The model has no parameters

Correct Answer: B. Training performance improves while validation performance worsens

Question 9

What does backpropagation primarily compute?

A. A test-set threshold

B. Parameter gradients using the chain rule

C. A new training dataset

D. The number of classes

Correct Answer: B. Parameter gradients using the chain rule

Question 10

Why is chronological splitting important in predictive-maintenance data?

A. It guarantees every class is balanced

B. It prevents future information from leaking into training

C. It makes ReLU unnecessary

D. It forces the output to be linear

Correct Answer: B. It prevents future information from leaking into training