Forward Propagation and Representation Learning

Introduction

A neural network transforms an input into an output through a sequence of computational operations. In a feed-forward network, this transformation is called forward propagation. The network receives numerical input features, combines them using learned weights and biases, applies nonlinear activation functions, and produces predictions. Although the calculation proceeds from input to output, its significance is deeper than simple arithmetic: each layer can learn a different representation of the data.

For example, an image-classification network may first respond to edges, then to corners and textures, and finally to object parts and complete objects. In a speech system, early layers may represent short acoustic patterns while deeper layers represent phonemes or words. This progressive construction of useful features is called representation learning.

Forward propagation is therefore central to both the operation and interpretation of deep learning systems. It is used during training to calculate predictions and during deployment to produce outputs for new data. This chapter develops the mathematical foundations of forward propagation, explains hierarchical feature learning, and applies the ideas to a practical classification problem.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the roles of weights, biases, layers, and activation functions.
2. Calculate a forward pass through one or more neural-network layers.
3. Distinguish linear transformations from nonlinear transformations.
4. Explain how multilayer networks learn hierarchical representations.
5. Interpret logits, probabilities, and classification outputs.
6. Identify common numerical and architectural problems in forward propagation.
7. Relate representation learning to practical AI applications.

1. The Basic Computational Unit

A neuron, or artificial unit, receives input values and calculates a weighted sum. For an input vector x = (x₁, x₂, …, xₙ), the pre-activation value is

Here, w is the weight vector, b is a scalar bias, and z is the pre-activation or weighted input. The weight wᵢ determines how strongly feature xᵢ influences the unit. The bias shifts the result independently of the input. The expression is dimensionally consistent when each product wᵢxᵢ has the same numerical interpretation and the bias has the same units as the weighted sum.

The neuron then applies an activation function:

The output a is passed to the next layer. Without an activation function, a sequence of neural-network layers would remain equivalent to one linear transformation. Nonlinear activations allow networks to model curved decision boundaries and complex relationships.

1.1 Common activation functions

The sigmoid and tanh functions can saturate: very large positive or negative inputs produce derivatives close to zero. ReLU avoids saturation on its positive side but can produce inactive units when inputs remain negative. The choice of activation depends on the task and network architecture.

2. Layers and Matrix Formulation

A layer containing several neurons can be written compactly using matrices. Suppose a layer receives an input vector a⁽ˡ⁻¹⁾ and produces an output a⁽ˡ⁾. The forward equations are

The superscript ⁽ˡ⁾ identifies layer l, not exponentiation. If layer l has nₗ units and the previous layer has nₗ₋₁ units, then W⁽ˡ⁾ has shape nₗ × nₗ₋₁, b⁽ˡ⁾ has shape nₗ, and a⁽ˡ⁾ has shape nₗ. The activation function is normally applied element by element, except that softmax couples all output logits through its denominator.

For a network with L layers, the complete forward pass is the composition

This expression shows that a deep network is a nested function. Each layer receives the representation constructed by the preceding layer.

Figure: Forward propagation through a multilayer network

The figure illustrates that each layer transforms the representation produced by the previous layer.

3. A Worked Forward-Pass Example

Consider a two-input network with one hidden layer of two ReLU units and one sigmoid output unit. Let

x = [1, 2]ᵀ

The hidden-layer parameters are

W⁽¹⁾ = [[1, −1], [0.5, 2]], b⁽¹⁾ = [0, 1]ᵀ

and the output-layer parameters are

W⁽²⁾ = [1, −2]], b⁽²⁾ = 0.5.

The brackets indicate vectors and matrices; the first matrix has two rows and two columns. We use ReLU in the hidden layer and sigmoid in the output layer.

Step 1: Hidden pre-activation

The first hidden unit receives

z₁⁽¹⁾ = 1(1) + (−1)(2) + 0 = −1.

The second receives

z₂⁽¹⁾ = 0.5(1) + 2(2) + 1 = 5.5.

Thus,

z⁽¹⁾ = [−1, 5.5]ᵀ.

Step 2: Hidden activation

Applying ReLU element by element gives

a⁽¹⁾ = [ReLU(−1), ReLU(5.5)]ᵀ = [0, 5.5]ᵀ.

Step 3: Output pre-activation

Step 4: Output probability

The sigmoid output is

The model therefore assigns a very small probability to the positive class. This is a forward pass only; it does not establish whether the prediction is correct. Correctness requires comparison with the known target label.

4. From Logits to Predictions

The output of a model depends on the task.

Binary classification

For binary classification, the final unit commonly produces a logit z and converts it to a probability using sigmoid:

A decision threshold, often 0.5, converts the probability into a class label. The threshold should not be treated as universally optimal; it may be adjusted when false positives and false negatives have different costs.

Multiclass classification

For K mutually exclusive classes, the output layer produces logits z₁, …, z_K. Softmax converts them into probabilities:

Every pⱼ lies between 0 and 1, and the probabilities sum to 1. The predicted class is usually the index with the largest probability. For numerical stability, implementations often subtract the largest logit m before exponentiation:

This produces the same probabilities because the common factor eᵐ cancels, while reducing the risk of overflow.

Regression

For regression, the output may be linear:

A linear output is appropriate when predictions can take any real value, such as temperature or demand. If the target has a restricted range, a suitable output transformation may be used.

5. Why Nonlinearity Matters

Suppose two layers contain only linear transformations:

Substitution gives

This is still a single affine transformation of x. Consequently, adding many linear layers without nonlinear activations does not create a genuinely deeper function. ReLU, sigmoid, tanh, or another nonlinear operation breaks this reducibility.

Nonlinear layers allow a network to combine simple conditions. For example, a hidden unit may respond when an input lies on one side of a boundary. Several hidden units can then combine these responses to construct regions with bends, corners, and more complex shapes. In this way, a network can represent functions that a single linear model cannot represent efficiently.

6. Representation Learning and Hierarchical Features

A representation is a numerical description of an input that makes useful information easier to detect. In a neural network, the activation vector at a hidden layer is a learned representation. The network does not usually receive explicit instructions such as “detect an edge” or “measure syllable duration.” Instead, training adjusts parameters so that intermediate representations help reduce prediction error.

6.1 Hierarchical composition

Deep networks learn through composition:

• The input layer contains raw or engineered measurements.
• Early hidden layers learn relatively local or simple combinations.
• Intermediate layers combine these into motifs, parts, or temporal patterns.
• Deeper layers form task-relevant concepts.
• The output layer maps the final representation to a prediction.

In image models, early convolutional layers often detect oriented edges and colour contrasts. Later layers may respond to textures, contours, eyes, wheels, or other object parts. In language models, representations may encode word relationships, syntactic patterns, semantic context, and task-specific information. These examples describe common tendencies rather than fixed rules: what a layer learns depends on data, architecture, optimization, and the objective function.

6.2 Distributed representations

A concept is usually represented across many units rather than by one unit alone. Such a distributed representation is efficient because combinations of activations can encode many patterns. It may also be robust: partial changes in the input can alter some activations without destroying the entire representation.

6.3 Invariance and equivariance

A useful representation may be invariant to irrelevant changes. For example, an image classifier should often recognize an object despite small changes in lighting. In other settings, the representation should be equivariant: a spatial shift in the input may produce a corresponding shift in an intermediate feature map. Convolutional networks deliberately exploit local structure and shared weights to support these properties.

6.4 Representation quality

A representation is useful when it supports the intended task, preserves relevant information, and suppresses irrelevant variation. It should not be judged only by visual inspection. Evaluation may include predictive accuracy, robustness tests, transfer to another task, calibration, fairness analysis, and sensitivity to perturbations.

7. Training and the Role of Forward Propagation

Forward propagation produces predictions. Training additionally requires a loss function that measures disagreement between the prediction and target. For a single binary example with target y ∈ {0,1} and predicted probability p, binary cross-entropy is

The logarithm is natural, and the formula assumes 0 < p < 1. In practice, stable implementations often combine the sigmoid and loss calculations to avoid taking the logarithm of a rounded zero.

For a batch of m examples, the mean loss is

Forward propagation calculates each p⁽ⁱ⁾ and therefore supplies the values needed by the loss. Backpropagation then calculates how changing each parameter would change the loss. An optimizer uses those gradients to update weights and biases. Thus, forward propagation is not the whole training process, but it is the necessary first computational stage.

A simplified parameter update is

where θ denotes a parameter, η > 0 is the learning rate, and ∇_θJ is the derivative of the batch loss with respect to that parameter. The update equation belongs to optimization, while the forward equations define the model’s prediction function.

8. Batch Processing and Tensor Shapes

Professional implementations process many examples simultaneously. If a batch contains m examples and each input has n features, a common convention stores the input matrix X with shape m × n. Depending on the software framework, weights may be arranged as nₗ₋₁ × nₗ rather than nₗ × nₗ₋₁. Both conventions are valid if matrix multiplication is consistent.

Shape errors are among the most common implementation problems. Before calculating a layer, verify:

1. The last dimension of the input matches the input dimension expected by the weight matrix.
2. The bias can be broadcast across all examples.
3. The activation is applied along the intended dimension.
4. The output shape matches the next layer’s input requirement.

For example, with X of shape m × n, weights W of shape n × h, and bias b of shape h, a batch layer may be written as

where Z has shape m × h. This is the row-wise equivalent of the column-vector notation introduced earlier.

9. Applied Case: Image-Based Quality Inspection

Case status: illustrative scenario. Consider a manufacturing company that inspects metal components using camera images. The objective is to classify each component as acceptable or defective.

Background

Manual inspection is slow and can vary between workers. The company collects labelled images, each associated with a quality decision. Images are converted into numerical arrays and normalized so that pixel values have a controlled scale.

Problem

A simple linear classifier performs well on obvious defects but fails when defects vary in position, orientation, and shape. The company needs a model that can learn useful visual representations rather than relying on manually designed rules.

Analysis

A multilayer network can first transform local pixel patterns into intermediate features. A convolutional architecture is particularly suitable because nearby pixels are related and the same defect pattern may appear at different locations. Early filters may respond to edges and boundaries. Deeper layers can combine these responses into scratches, dents, or irregular contours. The final layer produces a defect probability.

The forward process can be summarized as follows:

1. The normalized image is supplied as input.
2. Convolution and activation layers calculate local feature maps.
3. Pooling or strided operations reduce spatial resolution while retaining salient information.
4. Deeper layers combine feature maps into a compact representation.
5. A classification head produces a logit and sigmoid probability.
6. A threshold converts the probability into an operational decision.

Approach and solution

The development team divides data into training, validation, and test sets. Images from the same physical component must not appear in multiple sets, because that would create leakage. The team monitors not only accuracy but also recall for defective items, since missing a defect may be more costly than rejecting an acceptable item.

During training, the forward pass produces predictions, the loss compares them with labels, and backpropagation updates parameters. Data augmentation, such as small rotations or brightness changes, may improve robustness if those changes reflect real operating conditions. However, augmentation must not create unrealistic examples or remove the visual evidence of a defect.

Outcome

The model improves detection of varied defects, but its deployment is conditional. Images with poor lighting are flagged for human review, and confidence thresholds are selected using the costs of false acceptance and false rejection. The company retains human oversight and periodically audits performance across different machines, shifts, materials, and lighting conditions.

Learning points

• Representation learning can reduce dependence on manually specified visual rules.
• The architecture should reflect the structure of the data.
• A numerical probability is not automatically a reliable confidence estimate.
• Data leakage can produce deceptively strong test results.
• Operational decisions require cost-sensitive evaluation and monitoring.

10. Common Problems and Best Practices

Poor feature scaling

Large differences in input scale can make optimization difficult and cause some features to dominate weighted sums. Standardization or suitable normalization can help. The transformation must be estimated from training data and then applied consistently to validation, test, and production data.

Saturation and inactive units

Sigmoid and tanh units can saturate for large absolute inputs. ReLU units can become permanently inactive if their pre-activations remain negative. Appropriate initialization, normalization, learning rates, and activation choices reduce these risks.

Numerical overflow

Directly calculating eᶻ for very large z may overflow. Stable softmax implementations subtract the maximum logit. Stable loss functions similarly combine operations algebraically rather than calculating fragile intermediate values.

Overconfidence

A softmax or sigmoid output is a model score interpreted as probability under the model, not a guarantee. Calibration methods, reliability diagrams, and threshold analysis are useful when decisions depend on trustworthy probabilities.

Data and representation bias

A network may learn correlations that reflect historical or sampling bias. Representation learning does not automatically remove discrimination. Teams should examine subgroup performance, data coverage, label quality, and the consequences of errors.

Reproducibility and governance

Record the data version, preprocessing steps, parameter shapes, software environment, random seeds where relevant, and evaluation protocol. In regulated or safety-sensitive settings, maintain traceability, access controls, audit logs, and procedures for human review and model retirement.

11. Contemporary Developments

Modern architectures extend the basic forward-pass idea while preserving its essential structure. Residual networks add skip connections so that a layer can learn a correction to an earlier representation. Attention mechanisms calculate context-dependent combinations of representations. Transformers repeatedly apply projections, attention, nonlinear transformations, and normalization. Multimodal systems create representations for different data types and align them in shared or interacting spaces.

These developments do not eliminate the need to understand elementary forward propagation. Instead, they build on it. Every sophisticated architecture still performs a sequence of parameterized transformations, nonlinear operations, and output calculations. Understanding shapes, numerical stability, learned representations, and task-specific outputs remains essential for implementation and evaluation.

Chapter Summary

Forward propagation is the process by which a neural network transforms input data into a prediction. Each layer performs an affine transformation, z⁽ˡ⁾ = W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾, followed, in most hidden layers, by a nonlinear activation. Matrix notation makes multilayer calculations efficient and clarifies the required tensor shapes.

Nonlinearity is essential because a sequence of purely linear layers collapses into one affine transformation. Nonlinear activations enable complex decision boundaries and allow networks to construct hierarchical representations. Early layers commonly learn relatively simple patterns, while deeper layers combine them into increasingly task-relevant features.

The output transformation depends on the task: sigmoid is common for binary classification, softmax for mutually exclusive multiclass classification, and a linear output for unrestricted regression. Forward propagation supplies predictions to the loss function during training; backpropagation and optimization then adjust the parameters.

Reliable applications require careful preprocessing, shape checking, numerical stability, validation without data leakage, calibration, fairness assessment, and operational monitoring. Representation learning is powerful, but the learned features reflect both the data and the objective used to train the model.

Key Terms

Forward propagation: The computation of a network output by passing data from input to output.

Neuron: A computational unit that forms a weighted sum, adds a bias, and applies an activation.

Weight: A learned parameter that controls the influence of an input connection.

Bias: A learned additive parameter that shifts a pre-activation.

Pre-activation: The value produced before an activation function is applied.

Activation function: A function that transforms a neuron’s pre-activation, usually introducing nonlinearity.

Affine transformation: A linear mapping plus an additive bias.

Logit: An unnormalized output score before conversion to a probability.

Sigmoid: An activation that maps a scalar to the interval (0, 1).

ReLU: The rectified linear unit, defined as max(0, z).

Softmax: A function that converts multiple logits into probabilities summing to one.

Representation: A numerical encoding of input information.

Representation learning: Learning useful features automatically from data.

Hierarchical feature: A feature formed by combining lower-level features in a deeper layer.

Distributed representation: Information encoded across a pattern of multiple unit activations.

Logit stability: Numerical practices that prevent overflow or underflow during output computation.

Batch: A group of training examples processed together.

Data leakage: The unintended use of information from evaluation data during training or preprocessing.

Calibration: The agreement between predicted probabilities and observed frequencies.

Inference: Using a trained model to produce outputs for new data.

Review Questions

Short Answer Questions

1. What roles do weights and biases play in a neuron?
2. Why is a nonlinear activation needed between linear layers?
3. What is the difference between a logit and a probability?
4. When is softmax generally used?
5. What is meant by hierarchical representation learning?

Long Answer Questions

1. Derive the expression showing why two affine layers without nonlinear activations are equivalent to one affine transformation.
2. Explain the complete forward pass through a network with one hidden layer, including matrix dimensions.
3. Compare sigmoid, tanh, ReLU, and softmax in terms of definition and typical use.
4. Discuss how representations in image-processing networks may change from early to late layers.
5. Explain the relationship between forward propagation, loss calculation, backpropagation, and parameter updates.

Analytical / Application Questions

1. A network has input x = [2,1]ᵀ, hidden weights W⁽¹⁾ = [[1,2], [−1,1]], hidden bias b⁽¹⁾ = [0,1]ᵀ, and ReLU activation. Calculate the hidden activation.
2. A three-class classifier produces logits [2, 1, 0]. Explain which class is selected and why subtracting the maximum logit before softmax does not change the probabilities.
3. Design an evaluation plan for a defect-detection model in which missing a defective item is five times more costly than rejecting an acceptable item. Identify suitable metrics, threshold considerations, and governance controls.

Multiple Choice Questions with Answers

Question 1

Which equation correctly describes a layer’s pre-activation?

A. z⁽ˡ⁾ = W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾

B. z⁽ˡ⁾ = W⁽ˡ⁾ + a⁽ˡ⁻¹⁾b⁽ˡ⁾

C. z⁽ˡ⁾ = f⁽ˡ⁾(W⁽ˡ⁾a⁽ˡ⁻¹⁾)

D. z⁽ˡ⁾ = a⁽ˡ⁻¹⁾/W⁽ˡ⁾

Correct Answer: A. z⁽ˡ⁾ = W⁽ˡ⁾a⁽ˡ⁻¹⁾ + b⁽ˡ⁾

Question 2

What is the principal purpose of a nonlinear activation in a hidden layer?

A. To guarantee zero training loss

B. To make a sequence of layers equivalent to one linear layer

C. To enable the network to model nonlinear relationships

D. To remove the need for weights

Correct Answer: C. To enable the network to model nonlinear relationships

Question 3

For binary classification, which output function commonly converts a logit into a value between 0 and 1?

A. ReLU

B. Sigmoid

C. Softmax over unrelated batches

D. Linear scaling only

Correct Answer: B. Sigmoid

Question 4

What does softmax produce for a set of multiclass logits?

A. Values that sum to one

B. Only negative values

C. A single unconstrained real number

D. Binary labels without probabilities

Correct Answer: A. Values that sum to one

Question 5

In the worked example, what is the hidden activation after applying ReLU to z⁽¹⁾ = [−1,5.5]ᵀ?

A. [−1, 5.5]

B. [1, −5.5]

C. [0, 5.5]

D. [1, 5.5]

Correct Answer: C. [0, 5.5]

Question 6

Why can subtracting the largest logit improve softmax computation?

A. It changes the predicted class

B. It reduces the risk of exponential overflow without changing probabilities

C. It forces every probability to zero

D. It removes the need for normalization

Correct Answer: B. It reduces the risk of exponential overflow without changing probabilities

Question 7

Which statement best describes representation learning?

A. Manually writing every feature rule before training

B. Learning useful numerical features as part of model training

C. Using only the raw labels as inputs

D. Replacing all hidden layers with a lookup table

Correct Answer: B. Learning useful numerical features as part of model training

Question 8

What is data leakage?

A. A small numerical rounding error

B. Using evaluation information during training or preprocessing

C. A model producing a calibrated probability

D. A normal activation-function calculation

Correct Answer: B. Using evaluation information during training or preprocessing

Question 9

A linear output is often suitable for which task?

A. Unrestricted real-valued regression

B. Mutually exclusive class probability normalization

C. Converting every input to zero

D. Ensuring all predictions lie between zero and one

Correct Answer: A. Unrestricted real-valued regression

Question 10

What is the main function of forward propagation during training?

A. To calculate predictions used by the loss

B. To delete the model’s weights

C. To select the validation set randomly after testing

D. To guarantee fairness without evaluation

Correct Answer: A. To calculate predictions used by the loss