2. Mathematical and Computational Foundations

Introduction and Context

Neural networks are often introduced through diagrams of layers, nodes, and connections. Such diagrams are useful, but they conceal the mathematical operations that make learning possible. A neural network is, fundamentally, a parameterized function that transforms numerical inputs into outputs. Training adjusts its parameters so that the function produces useful predictions. Linear algebra represents data and parameters; calculus explains learning through derivatives; probability and statistics describe uncertainty and data variation; information theory motivates several loss functions; and numerical computation makes the entire process executable on digital hardware.

For postgraduate study, it is not sufficient to memorize formulas. One must understand the shape of every quantity, the assumptions behind each operation, the direction in which information flows, and the numerical conditions under which an algorithm succeeds or fails. A small error in tensor dimensions, derivative calculation, or numerical scaling can invalidate an otherwise sophisticated model.

This chapter develops these foundations as an integrated framework. It begins with vectors, matrices, and tensors, then examines functions and derivatives, probability-based modeling, loss functions, backpropagation, optimization, and computational practice. The emphasis is on the reasoning required to implement, inspect, and diagnose neural-network systems.

Learning Objectives

After studying this chapter, the learner should be able to:

1. Represent neural-network data and parameters using vectors, matrices, and tensors.

2. Explain dot products, matrix multiplication, norms, rank, broadcasting, and tensor shapes.

3. Use partial derivatives, gradients, Jacobians, and the chain rule in network computations.

4. Relate probability distributions and statistical estimation to neural-network outputs.

5. Explain the construction and behavior of common loss functions.

6. Derive the essential steps of backpropagation for layered models.

7. Apply gradient-based optimization while recognizing numerical and conditioning problems.

8. Diagnose common implementation errors involving dimensions, scaling, initialization, and floating-point arithmetic.

9. Interpret a practical neural-network training problem using mathematical and computational evidence.

1. Neural Networks as Parameterized Functions

Let an input be represented by a vector x and a target by y. A neural network defines a function

f(x; θ),

where θ denotes all trainable parameters, typically weights and biases. The training problem is to choose θ so that the predictions f(x; θ) agree with observed targets. This is expressed through an objective such as

J(θ) = (1/n) Σᵢ L(f(xᵢ; θ), yᵢ),

where L is a loss function and n is the number of training examples.

A single dense layer computes

z = Wx + b,

a = φ(z),

where W is a matrix, b is a bias vector, φ is an activation function, and a is the layer output. A multilayer network composes these transformations:

f(x; θ) = φₗ(Wₗ φₗ₋₁(...φ₁(W₁x + b₁)...)+bₗ).

This composition explains why local mathematical operations matter. Every layer must have compatible dimensions, differentiable or subdifferentiable operations, and numerically stable values.

2. Linear Algebra: The Language of Network Computation

Vectors, matrices, and tensors

A scalar is a single number. A vector is an ordered one-dimensional collection, commonly written x ∈ Rᵈ. A matrix is a rectangular array, such as W ∈ Rᵐˣᵈ. A tensor is a generalization to any number of dimensions. In machine learning, an image may be represented as a height × width × channel tensor, while a sequence batch may have shape batch × time × features.

Shape is not merely descriptive metadata. It determines whether an operation is mathematically defined. If x ∈ Rᵈ and W ∈ Rᵐˣᵈ, then Wx ∈ Rᵐ. If a batch X has shape n × d and examples are stored in rows, the equivalent dense-layer operation is XWᵀ + b, producing shape n × m. Confusing row and column conventions is a frequent source of errors.

Inner products and norms

The inner product of two vectors is

xᵀy = Σⱼ xⱼyⱼ.

It measures alignment and is central to linear prediction. The Euclidean norm is ||x||₂ = √(Σⱼxⱼ²), while the squared norm ||x||₂² is often used because it avoids a square root. The L₁ norm, ||x||₁ = Σⱼ|xⱼ|, encourages sparsity when used as a regularizer.

Norms quantify magnitude. Large parameter or activation magnitudes can produce unstable gradients, whereas excessively small magnitudes can make learning slow. Regularization terms such as λ||W||₂² constrain parameter magnitude and may improve generalization.

Matrix multiplication and linear transformations

Matrix multiplication is not elementwise multiplication. For A ∈ Rᵐˣᵖ and B ∈ Rᵖˣⁿ, the product AB ∈ Rᵐˣⁿ, with entries

(AB)ᵢⱼ = Σₖ AᵢₖBₖⱼ.

A matrix represents a linear transformation: it can rotate, scale, reflect, project, or combine coordinates. A bias makes an affine transformation rather than a strictly linear one. The activation function is what enables a network composed of layers to represent nonlinear relationships.

Rank describes the dimension of the space spanned by a matrix's rows or columns. Low-rank structure can reduce computational cost and parameter count, but an overly restrictive rank may limit representational capacity. Singular values reveal directions that are strongly or weakly amplified and are relevant to conditioning and gradient propagation.

Broadcasting and tensor operations

Numerical libraries frequently permit broadcasting: a lower-dimensional array is conceptually replicated to match a larger shape. In XWᵀ + b, a bias vector is added to every row. Broadcasting improves efficiency, but it can hide errors. For example, an unintended shape of (m, 1) may produce a matrix of unexpected dimensions rather than a clear failure. Professional implementations therefore inspect shapes and use assertions or typed tensor specifications.

Transpose, reshape, concatenate, reduction, and convolution are additional fundamental operations. A reshape changes interpretation without changing the number of elements; it does not by itself rearrange values. A reduction such as mean or sum removes one or more dimensions and influences gradient scale. These distinctions become important when moving between individual examples and batches.

3. Calculus and the Geometry of Learning

Derivatives and partial derivatives

For a scalar function y = f(x), the derivative measures the local rate of change. In several variables, a scalar-valued function J(w₁,...,wₚ) has partial derivatives with respect to each parameter. The gradient is

∇J = [∂J/∂w₁, ..., ∂J/∂wₚ]ᵀ.

The gradient points in the direction of greatest local increase. Therefore, gradient descent updates parameters in the opposite direction:

θₜ₊₁ = θₜ − η∇J(θₜ),

where η is the learning rate.

Chain rule and computational graphs

If y = f(g(x)), then dy/dx = (dy/dg)(dg/dx). Neural networks contain many nested functions, so derivatives are calculated by applying the chain rule backward through a computational graph. Each node stores values from the forward pass and propagates an appropriate derivative during the backward pass.

Consider a scalar unit:

z = wx + b, a = φ(z), L = ½(a − y)².

Then

∂L/∂a = a − y,

∂a/∂z = φ′(z),

∂z/∂w = x,

and consequently

∂L/∂w = (a − y)φ′(z)x.

This expression shows that a gradient is a product of local factors. If one factor is near zero, the entire gradient can become small; if factors are large, it can become unstable.

Jacobians and vector-valued functions

When a vector output depends on a vector input, the derivative is a Jacobian matrix. If y ∈ Rᵐ and x ∈ Rᵈ, the Jacobian has entries Jᵢⱼ = ∂yᵢ/∂xⱼ. In practical automatic differentiation, explicit Jacobian matrices are often avoided because vector-Jacobian or Jacobian-vector products are more efficient.

The Hessian, the matrix of second derivatives, describes local curvature. Although full Hessians are expensive for large networks, curvature affects optimization: a narrow valley may require different step sizes in different directions. Second-order methods and adaptive optimizers approximate or exploit such information.

Activation functions

The sigmoid function σ(z) = 1/(1+e⁻ᶻ) maps real values to (0,1), making it useful for binary probabilities. Its derivative is σ(z)(1−σ(z)); for highly positive or negative z, this derivative approaches zero. The hyperbolic tangent maps to (−1,1) but also saturates.

The rectified linear unit, ReLU(z)=max(0,z), has derivative 1 for positive inputs and 0 for negative inputs, apart from the convention at zero. It is computationally simple and often reduces saturation, although units that remain negative may become inactive. Variants such as leaky ReLU provide a small negative-side slope. Softmax converts logits z into a categorical probability vector:

softmax(z)ᵢ = exp(zᵢ − c) / Σⱼ exp(zⱼ − c),

where c = maxⱼ zⱼ is subtracted for numerical stability.

4. Probability, Statistics, and Information Theory

Random variables and distributions

A random variable represents uncertain outcomes. A probability mass function describes discrete outcomes, while a density describes continuous values. A conditional model p(y|x;θ) expresses the distribution of a target given an input. Neural networks commonly parameterize this distribution rather than directly predicting a single deterministic value.

For binary classification, the network may output p = σ(z), interpreted as the probability of class 1. For multiclass classification, softmax outputs probabilities that sum to one. For regression, a network may predict the conditional mean, or both mean and variance when uncertainty is modeled explicitly.

Likelihood and maximum likelihood estimation

Given independent observations, the likelihood is

p(y₁,...,yₙ|X;θ) = Πᵢ p(yᵢ|xᵢ;θ).

Maximizing likelihood is equivalent to minimizing negative log-likelihood:

−log Πᵢp(yᵢ|xᵢ;θ) = −Σᵢ log p(yᵢ|xᵢ;θ).

The logarithm converts products into sums and improves numerical handling. It also produces familiar loss functions. Bernoulli negative log-likelihood is binary cross-entropy; categorical negative log-likelihood is categorical cross-entropy. Gaussian assumptions lead to squared-error terms, with the precise form depending on whether variance is fixed or learned.

Entropy, cross-entropy, and divergence

The entropy of a discrete distribution p is H(p)=−Σᵢpᵢlog pᵢ. It measures uncertainty, with the logarithm base determining the unit. Cross-entropy between a target distribution p and model distribution q is H(p,q)=−Σᵢpᵢlog qᵢ. The Kullback–Leibler divergence is

D_KL(p||q)=Σᵢpᵢlog(pᵢ/qᵢ),

which is nonnegative but not generally symmetric. Cross-entropy equals H(p)+D_KL(p||q), so minimizing it encourages the model distribution to approach the target distribution.

Statistics and generalization

A sample is used to estimate properties of a population. The empirical risk is the average loss on observed data; expected risk concerns the underlying data-generating distribution. Training performance alone does not establish generalization. Validation and test sets provide evidence about performance on unseen data, provided that data leakage is avoided.

Mean and variance are sensitive to outliers, while robust summaries such as the median may be useful for preprocessing. Standardization commonly uses x′=(x−μ)/σ, with μ and σ computed from the training set and then applied unchanged to validation and test data. Computing them using all data leaks information from evaluation sets.

5. Loss Functions and Their Interpretation

A loss function converts prediction error into a scalar quantity suitable for optimization. Mean squared error is

MSE = (1/n)Σᵢ(ŷᵢ−yᵢ)².

It strongly penalizes large errors and corresponds to a Gaussian-noise interpretation under common assumptions. Mean absolute error uses |ŷ−y| and is less sensitive to outliers, though it is not differentiable at zero; subgradients or smooth approximations can be used.

Binary cross-entropy is

−[y log p +(1−y)log(1−p)].

It penalizes confident incorrect predictions heavily. For numerical stability, implementations often combine logits and the loss directly rather than first computing probabilities and then taking logarithms. In multiclass classification, cross-entropy uses the target class's log probability. Label smoothing replaces a hard target distribution with a slightly softened one and can reduce excessive confidence, but it changes the learning objective.

Loss choice is a modeling decision, not merely a software setting. It should reflect the target type, error costs, uncertainty assumptions, and evaluation objectives. Class imbalance may require weighting, resampling, or an alternative objective, but such choices should be validated rather than applied mechanically.

6. Backpropagation and Optimization

Backpropagation is an efficient application of the chain rule. During the forward pass, the network computes predictions and stores intermediate activations. During the backward pass, it starts with the derivative of the loss and propagates gradients toward earlier layers. For a dense layer with z = Wa+b and upstream gradient δ = ∂L/∂z,

∂L/∂W = δaᵀ,

∂L/∂b = δ,

∂L/∂a = Wᵀδ.

For a batch, these expressions are aggregated according to the chosen reduction, usually a sum or mean. The distinction matters: a sum makes gradient magnitude grow with batch size, whereas a mean generally keeps its scale more comparable across batch sizes.

Stochastic gradient descent estimates the gradient using a minibatch:

θ ← θ − η ĝ.

Momentum maintains a moving direction and can reduce oscillation. Adaptive methods such as Adam maintain estimates of first and second moments of gradients and adjust parameter-wise step sizes. These methods are effective but do not eliminate the need for learning-rate selection, monitoring, and validation.

Regularization may be introduced through an objective such as

J_reg = J + λΩ(θ),

where Ω may be an L₂ penalty, an L₁ penalty, or another constraint. Early stopping, data augmentation, dropout, and architectural restrictions also influence effective model complexity. Their value depends on the data and task rather than on universal guarantees.

7. Numerical Computation in Practice

Neural-network calculations use finite-precision arithmetic. Floating-point numbers cannot represent every real number, and operations may incur rounding error, overflow, underflow, or cancellation. Exponentials in softmax can overflow if logits are large; the max-subtraction trick prevents this in ordinary circumstances. Logarithms of zero are undefined, so stable implementations avoid explicitly forming probabilities when a logit-based expression is available.

Conditioning describes how sensitive a problem's output is to small input changes. Poorly scaled features can create ill-conditioned optimization landscapes. Standardization, suitable initialization, normalization methods, and careful learning-rate schedules can improve conditioning.

Initialization affects signal and gradient propagation. If all weights are initialized identically, units receive identical gradients and remain equivalent. Random initialization breaks this symmetry. Xavier-type initialization is associated with maintaining variance across layers for certain activations, while He-type initialization is designed for rectifier-like activations. These are starting principles, not substitutes for diagnostics.

Gradient clipping limits unusually large gradients, either by elementwise thresholding or by rescaling the whole gradient when its norm exceeds a limit. It is particularly useful in some recurrent or unstable optimization settings, but excessive clipping can conceal deeper problems. Monitoring training and validation loss, gradient norms, activation statistics, and parameter magnitudes provides evidence about what is happening inside the model.

Automatic differentiation reduces derivative implementation errors, but it does not guarantee a correct model. Gradient checking compares analytical or automatic derivatives with finite-difference approximations on a small test case. It is valuable for custom layers and unusual operations. Shape tests, deterministic seeds, unit tests for loss functions, and checks for NaN or infinite values are equally important.

8. An Illustrative Applied Case: Predictive Maintenance

Background

Consider a fictional manufacturing plant that wishes to predict whether a machine will require maintenance within the next seven days. Sensors record temperature, vibration, pressure, and operating duration at regular intervals.

Situation and problem

The original data contain measurements on different scales, missing readings, and far fewer failure events than non-failure events. A team trains a dense neural network and observes high training accuracy but unstable validation performance.

Analysis

The input features are standardized using only training-set statistics. Missing values are handled through a documented imputation strategy, and the split is performed by time so that future observations do not enter the training set. Because failure is rare, accuracy is inadequate as the sole metric. The team examines precision, recall, calibration, and the operational cost of missed failures and unnecessary inspections.

The output layer contains one logit z, converted conceptually to p=σ(z). Rather than computing sigmoid followed by a logarithm manually, the implementation uses a numerically stable binary cross-entropy-with-logits loss. Class weighting is considered because false negatives are expensive, but its effect is evaluated on a validation period.

Approach and solution

The model uses a modest architecture with ReLU hidden units, He-style initialization, minibatch optimization, early stopping, and gradient monitoring. Baseline logistic regression and a tree-based model are retained for comparison. Threshold selection is separated from model training: the default threshold of 0.5 is not assumed to be operationally optimal.

Outcome

The final decision is based not only on the neural network's loss but on temporal validation results, calibration, maintenance capacity, and error costs. The case illustrates that mathematical foundations support organizational decisions: tensor shapes enable correct implementation, probability supports calibrated risk estimates, loss design reflects costs, and numerical monitoring protects reliability.

Learning points

The case demonstrates five principles. First, preprocessing is part of the statistical model. Second, a loss function encodes assumptions and priorities. Third, validation must reflect deployment conditions. Fourth, numerical stability is a practical requirement. Fifth, a neural network should be compared with simpler alternatives rather than treated as automatically superior.

Challenges, Risks, and Professional Considerations

Common errors include transposing a weight matrix incorrectly, averaging losses over the wrong dimension, applying inconsistent preprocessing, interpreting logits as probabilities, and evaluating on leaked data. Vanishing or exploding gradients may result from activation saturation, poor initialization, excessive depth, or unsuitable learning rates. Overfitting can arise when model capacity exceeds the information available in the data.

Numerical and statistical correctness do not guarantee ethical correctness. If a network supports safety, employment, lending, healthcare, or other consequential decisions, practitioners must document data provenance, uncertainty, intended use, limitations, and escalation procedures. Sensitive attributes and proxy variables require scrutiny. Reproducible experiments, versioned datasets, audit logs, access controls, and human review are governance practices rather than optional administrative tasks.

Emerging Directions and Best Practices

Modern neural-network systems increasingly use mixed-precision arithmetic, accelerator hardware, automatic differentiation, compiler-based graph optimization, and distributed training. These developments improve speed and energy efficiency but introduce additional concerns, including loss scaling, nondeterminism, communication overhead, and hardware-dependent reproducibility. Probabilistic prediction, uncertainty estimation, differentiable programming, and structured neural operators extend the mathematical role of neural networks beyond simple point prediction.

A sound workflow is to define the prediction target and evaluation criterion, inspect and split the data appropriately, specify tensor shapes, establish a simple baseline, select a loss consistent with the probabilistic task, standardize inputs using training data only, monitor both optimization and generalization, test numerical stability, and document every consequential decision. Mathematical transparency is especially important when models are deployed beyond the laboratory.

Chapter Summary

Neural networks are parameterized compositions of affine transformations and nonlinear functions. Vectors, matrices, and tensors provide the representation system, while matrix multiplication, broadcasting, reshaping, and reductions determine computational behavior. Derivatives, gradients, Jacobians, and the chain rule explain backpropagation and gradient-based learning. Probability and statistics connect outputs to distributions, likelihood, uncertainty, sampling, and generalization. Cross-entropy, squared error, and other losses encode both mathematical assumptions and practical priorities. Numerical computation introduces issues of finite precision, conditioning, initialization, stability, and reproducibility. Effective practice combines sound mathematics with shape checking, stable implementations, validation, monitoring, baselines, and responsible governance.

Key Terms

Scalar: A single numerical value.

Vector: An ordered one-dimensional array of values.

Matrix: A rectangular two-dimensional array representing, among other things, a linear transformation.

Tensor: A multidimensional numerical array used to represent data or parameters.

Broadcasting: An operation in which lower-dimensional data are virtually expanded to compatible shapes.

Norm: A function that measures the magnitude of a vector or tensor.

Rank: The dimension of the row or column space of a matrix.

Activation function: A function applied to a layer's pre-activation values to introduce nonlinear behavior.

Gradient: The vector of first partial derivatives of a scalar function with respect to its parameters.

Jacobian: The matrix of first derivatives of a vector-valued function.

Hessian: The matrix of second-order partial derivatives of a scalar function.

Backpropagation: Efficient reverse application of the chain rule through a computational graph.

Likelihood: The probability of observed data viewed as a function of model parameters.

Cross-entropy: A loss measuring the discrepancy between a target distribution and a predicted distribution.

Entropy: A measure of uncertainty in a probability distribution.

Kullback–Leibler divergence: A nonnegative, generally asymmetric measure of discrepancy between distributions.

Regularization: The use of penalties or constraints to control model complexity.

Conditioning: The sensitivity of a computational problem to small changes in its inputs.

Gradient clipping: Limiting or rescaling gradients to control unusually large updates.

Automatic differentiation: Programmatic calculation of derivatives by applying chain-rule operations to a computational graph.

Review Questions

Short Answer Questions

1. Distinguish a tensor's shape from its numerical values.

2. Why does a bias make a layer affine rather than strictly linear?

3. State the gradient-descent update rule and explain the role of the learning rate.

4. What is the relationship between maximum likelihood and negative log-likelihood?

5. Why is subtracting the maximum logit useful in softmax computation?

Long Answer Questions

1. Explain how matrix multiplication, activation functions, and composition produce a multilayer neural network.

2. Derive the gradient of a scalar neuron with squared-error loss using the chain rule.

3. Compare mean squared error, mean absolute error, and cross-entropy in terms of assumptions and behavior.

4. Describe the forward and backward passes of backpropagation for a dense layer.

5. Discuss the causes and remedies of vanishing and exploding gradients.

Analytical / Application Questions

1. A batch matrix X has shape 64 × 20 and a dense layer has 12 output units. Determine the required shape of W when computing XWᵀ and the output shape, including a bias vector.

2. A classifier achieves high accuracy on a dataset in which failures are rare. Using the predictive-maintenance case, explain why accuracy may be misleading and propose a validation strategy.

3. A training run produces NaN losses after several iterations. Develop a diagnostic sequence involving logits, learning rate, input scaling, gradients, and floating-point stability.

22. Multiple Choice Questions with Answers

Question 1

If X has shape 32 × 10 and a dense layer has 6 output units using XWᵀ + b, what is the shape of W?

A. 32 × 6

B. 10 × 6

C. 6 × 10

D. 10 × 32

Correct Answer: C. 6 × 10

Question 2

Why is the gradient used in the negative direction during gradient descent?

A. It points toward the direction of greatest local increase

B. It always points toward the global minimum

C. It removes the need for a learning rate

D. It converts a nonlinear function into a linear one

Correct Answer: A. It points toward the direction of greatest local increase

Question 3

Which operation most directly prevents overflow in a softmax calculation?

A. Dividing every logit by its class index

B. Subtracting the largest logit before exponentiation

C. Replacing all logits with zeros

D. Applying an L1 penalty to the logits

Correct Answer: B. Subtracting the largest logit before exponentiation

Question 4

For binary classification, binary cross-entropy is most naturally interpreted as the negative log-likelihood of which distribution?

A. Gaussian

B. Poisson

C. Bernoulli

D. Uniform

Correct Answer: C. Bernoulli

Question 5

If a loss is averaged over a minibatch rather than summed, what is the usual effect?

A. Its gradient scale is less dependent on batch size

B. All gradients become exactly zero

C. The model becomes convex

D. The batch contains fewer examples

Correct Answer: A. Its gradient scale is less dependent on batch size

Question 6

What is the main purpose of subtracting training-set means and dividing by training-set standard deviations?

A. To guarantee perfect accuracy

B. To improve feature scale and conditioning without leaking evaluation data

C. To convert labels into probabilities

D. To increase the number of model parameters

Correct Answer: B. To improve feature scale and conditioning without leaking evaluation data

Question 7

Why can identical initialization of all weights in a layer be problematic?

A. It makes every activation probabilistic

B. It prevents the bias from being learned

C. It causes units to receive identical updates and remain symmetric

D. It guarantees exploding gradients

Correct Answer: C. It causes units to receive identical updates and remain symmetric

Question 8

In a predictive-maintenance problem with rare failures, why might accuracy be inadequate as the only evaluation measure?

A. Accuracy cannot be computed for binary outcomes

B. A model can appear accurate while missing most costly failures

C. Accuracy is always lower than cross-entropy

D. Accuracy measures only numerical stability

Correct Answer: B. A model can appear accurate while missing most costly failures

Question 9

Which derivative object is most appropriate for a vector-valued output with respect to a vector input?

A. Scalar norm

B. Jacobian

C. Entropy

D. Likelihood

Correct Answer: B. Jacobian

Question 10

What does gradient clipping primarily attempt to control?

A. The number of classes

B. The dimensionality of the input tensor

C. Unusually large parameter updates caused by large gradients

D. The amount of validation data

Correct Answer: C. Unusually large parameter updates caused by large gradients