3. Feedforward Networks and Representation Learning

Introduction and Context

A feedforward neural network is one of the clearest settings in which the central ideas of deep learning become visible. Information enters through an input layer, is transformed by one or more intermediate layers, and produces an output without circulating through recurrent connections. Despite this apparently simple flow, feedforward networks can approximate highly complex functions when their architecture, activation functions, parameters, and training procedure are appropriately designed.

The importance of feedforward networks extends beyond their use as standalone classifiers or regressors. Their hidden layers learn representations: progressively transformed descriptions of the input that make prediction easier. A raw image, for example, may be represented first through local intensity patterns, then through edges and textures, and finally through object-level configurations. In tabular applications, hidden units may combine measured variables into nonlinear indicators of risk, demand, or eligibility. Representation learning therefore connects network architecture with the broader objective of discovering useful features from data rather than relying entirely on manually engineered variables.

This chapter develops the perceptron and multilayer perceptron (MLP), explains forward propagation and parameter initialization, and examines how activation functions and topology influence the representations a network can learn. It also considers practical design decisions, failure modes, responsible use, and an applied case involving an MLP for credit-risk screening.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain the structure and operation of a perceptron and an MLP.

2. Distinguish linear and nonlinear activation functions and select suitable output activations.

3. Formulate forward propagation mathematically and interpret its computational graph.

4. Explain why parameter initialization matters and compare common initialization strategies.

5. Analyze how depth, width, topology, and bottlenecks affect learned representations.

6. Apply practical principles for designing feedforward networks for classification and regression.

7. Identify common implementation errors, risks, and ethical issues in applied neural-network systems.

From a Linear Decision Rule to a Multilayer Model

The perceptron

For an input vector \(\mathbf{x}=[x_1,x_2,\ldots,x_d]^T\), a perceptron computes a weighted sum and bias:

\[

z=\mathbf{w}^T\mathbf{x}+b.

\]

A traditional binary perceptron applies a threshold function:

\[

\hat{y}=\begin{cases}1,&z\geq 0\\0,&z<0.\end{cases}

\]

The weights determine the contribution of each feature, while the bias shifts the decision boundary. The resulting boundary, \(\mathbf{w}^T\mathbf{x}+b=0\), is a hyperplane. Consequently, a single perceptron can solve linearly separable problems but cannot represent functions such as exclusive OR (XOR), whose classes cannot be separated by one hyperplane.

The perceptron is historically and conceptually important because it establishes the basic computational unit of a neural network. However, its hard threshold is discontinuous and therefore inconvenient for gradient-based optimization. Modern networks generally replace it with differentiable or almost-everywhere differentiable activations.

The multilayer perceptron

An MLP connects layers of computational units in one direction. A typical network contains an input representation, one or more hidden layers, and an output layer. For hidden layer \(l\), the transformation is

\[

\mathbf{a}^{(l)}=\phi^{(l)}(\mathbf{z}^{(l)}),\qquad

\mathbf{z}^{(l)}=\mathbf{W}^{(l)}\mathbf{a}^{(l-1)}+\mathbf{b}^{(l)},

\]

where \(\mathbf{a}^{(0)}=\mathbf{x}\), \(\mathbf{W}^{(l)}\) is a weight matrix, \(\mathbf{b}^{(l)}\) is a bias vector, and \(\phi^{(l)}\) is an element-wise activation function. The final output is produced by the same pattern, although its activation is chosen according to the task.

An MLP with no nonlinear hidden activation is still only a linear or affine mapping, regardless of how many layers it contains. Nonlinearity is therefore not an optional decoration; it is what permits composition of transformations to form curved decision boundaries and rich function approximations. The universal approximation result states, broadly, that a sufficiently wide network with an appropriate nonconstant activation can approximate a wide class of continuous functions on compact domains. This result does not imply that a small network will learn the function efficiently, generalize well, or optimize easily.

Activation Functions as Representation Shapers

An activation function controls how a unit transforms its pre-activation and consequently influences both the geometry of the learned representation and the flow of gradients.

Sigmoid and hyperbolic tangent

The sigmoid function is

\[

\sigma(z)=\frac{1}{1+e^{-z}}.

\]

It maps values to \((0,1)\), making it suitable for a binary output interpreted as a probability when paired with binary cross-entropy. In hidden layers, however, sigmoid units can saturate for large positive or negative inputs. Their derivatives become small, producing weak gradients. Sigmoid activations also are not zero-centered, which can make optimization less convenient.

The hyperbolic tangent,

\[

\tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}},

\]

maps values to \((-1,1)\) and is zero-centered. It can represent signed intermediate features, but it also saturates. Both sigmoid and tanh remain useful in particular architectures and output layers, but they are less common as default hidden activations in deep MLPs.

ReLU and its variants

The rectified linear unit (ReLU) is

\[

\operatorname{ReLU}(z)=\max(0,z).

\]

It is computationally simple, preserves a strong gradient for positive inputs, and often yields sparse activations because negative pre-activations become zero. A drawback is the “dying ReLU” problem: a unit whose pre-activations remain negative may stop receiving useful gradients and effectively become inactive.

Leaky ReLU assigns a small negative slope to negative inputs:

\[

\operatorname{LReLU}(z)=\max(\alpha z,z),

\]

where \(\alpha\) is small and positive. Parametric ReLU learns this slope. Other smooth or self-gated activations, such as GELU and Swish, are also used, especially when optimization quality and smoothness are important. No activation is universally best; selection should consider the architecture, initialization, data scale, and empirical validation.

Output activations

Output activation should match the statistical meaning of the target. Common choices include:

• Binary classification: one sigmoid output, usually trained with binary cross-entropy.
• Mutually exclusive multiclass classification: a softmax vector, \(p_k=e^{z_k}/\sum_j e^{z_j}\), trained with categorical cross-entropy.
• Multilabel classification: independent sigmoid outputs, because several labels may be true simultaneously.
• Unconstrained regression: a linear output.
• Positive regression, such as duration or nonnegative demand: a positive-valued output such as softplus, or a suitable transformation of the target.

A frequent design error is to use softmax for multilabel problems or sigmoid outputs for mutually exclusive classes without a coherent loss formulation.

Forward Propagation: Computation Through the Network

Forward propagation is the ordered evaluation of a network for a given input. Consider a two-hidden-layer MLP:

\[

\mathbf{z}^{(1)}=\mathbf{W}^{(1)}\mathbf{x}+\mathbf{b}^{(1)},\quad

\mathbf{a}^{(1)}=\phi_1(\mathbf{z}^{(1)}),

\]

\[

\mathbf{z}^{(2)}=\mathbf{W}^{(2)}\mathbf{a}^{(1)}+\mathbf{b}^{(2)},\quad

\mathbf{a}^{(2)}=\phi_2(\mathbf{z}^{(2)}),

\]

\[

\hat{\mathbf{y}}=g(\mathbf{W}^{(3)}\mathbf{a}^{(2)}+\mathbf{b}^{(3)}).

\]

The model’s parameters are all weights and biases. A loss function compares \(\hat{\mathbf{y}}\) with the target \(\mathbf{y}\). Forward propagation itself does not change parameters; it generates predictions and the intermediate values needed for backpropagation. In software, these operations are represented by a computational graph, which also enables automatic differentiation.

For a minibatch of \(m\) examples, matrix orientation and dimensions must be consistent. If inputs are arranged as rows, a dense layer may be written as \(\mathbf{Z}=\mathbf{A}\mathbf{W}+\mathbf{b}\), with broadcasting of the bias across rows. Explicit dimensional reasoning is a powerful debugging technique. If a layer has \(n_{l-1}\) inputs and \(n_l\) units, its weight matrix contains \(n_{l-1}n_l\) parameters and its bias contains \(n_l\), giving \(n_{l-1}n_l+n_l\) parameters in total.

Topology and the Formation of Internal Representations

Network topology describes how units and layers are connected. In a fully connected MLP, every unit in one layer connects to every unit in the next. Width determines the number of units in a layer; depth determines the number of layers containing transformations. A wider network can learn many features in parallel, while a deeper network can compose features across levels.

Depth is not merely a larger parameter count. A deep composition can represent some functions more economically than a shallow network, because later layers reuse and combine features formed earlier. For example, a hidden layer may encode interactions between standardized financial variables, and a later layer may combine those interactions into a probability of default. Yet excessive depth can make optimization more difficult and may increase sensitivity to initialization and regularization.

A narrow hidden layer can serve as a bottleneck. If its dimension is smaller than the input dimension, the network must preserve only information useful for its objective. Such bottlenecks may produce compact representations, but compression can also discard information needed for fairness analysis, calibration, or downstream tasks. Representation quality is therefore task-dependent: a feature space useful for predicting one target may encode spurious correlations or be unhelpful for another.

Skip connections, although associated especially with other neural architectures, can also be used in feedforward designs. They provide shorter paths for information and gradients and may reduce optimization difficulty. Residual-style topology is particularly useful when adding layers would otherwise make training unstable or cause degradation.

Parameter Initialization and Optimization Readiness

Initialization is the starting configuration of weights before learning. If all weights are initialized to the same value, units in the same layer receive identical signals and gradients; they remain symmetric and learn the same feature. Randomness is therefore needed to break symmetry, although biases may often be initialized to zero.

Poor scale is another danger. If weights are too large, activations and gradients may explode. If they are too small, signals and gradients may vanish as they pass through layers. Initialization schemes address this by relating the variance of weights to the number of incoming or outgoing connections.

For a layer with ReLU-like activations, He initialization commonly samples weights with variance approximately \(2/n_{in}\). For approximately linear or tanh-like activations, Xavier or Glorot initialization uses a variance related to \(2/(n_{in}+n_{out})\). These formulas are practical approximations rather than guarantees. They work best when features are appropriately scaled and the assumed activation behavior is reasonably satisfied.

Initialization should be evaluated through diagnostics: inspect activation distributions, gradient magnitudes, loss curves, and the fraction of inactive ReLU units. Reproducible experiments require recording random seeds and initialization settings, while serious model development should still evaluate multiple seeds because a single training run may be unrepresentative.

Representation Learning in Practice

Representation learning occurs when hidden layers discover transformations that make the target function easier to express. In supervised MLPs, this learning is guided by the prediction loss. The network is not necessarily learning a universally meaningful description of the data; it is learning a representation useful for the specified objective and training distribution.

Feature scaling is especially important for MLPs. Variables measured in very different ranges can produce badly conditioned optimization problems, causing some features to dominate early updates. Standardization using training-set statistics is common. The mean and variance must be calculated only from the training partition and then applied unchanged to validation and test data; otherwise, information leakage occurs.

Representations can be examined through activation statistics, sensitivity analysis, probing models, dimensionality-reduction visualizations, and perturbation tests. These methods are informative but should not be treated as definitive explanations. A two-dimensional visualization may obscure important directions, and a feature probe may reveal information encoded in a representation without showing that the original network relies on it causally.

Transfer learning with MLPs is less universally effective than with models designed for images or language, but learned embeddings and hidden representations can still be reused when source and target tasks share data-generating structure. Fine-tuning should be accompanied by checks for distribution shift and unintended transfer of sensitive information.

Designing an MLP: A Practical Workflow

A defensible design begins with the problem rather than with a preferred architecture. First define the prediction target, unit of analysis, decision horizon, and cost of errors. Next establish a simple baseline, such as logistic regression or a tree-based model. The baseline provides a performance reference and may be more interpretable or robust on small tabular data.

Prepare data using a leakage-resistant pipeline. Handle missing values, encode categorical variables, scale numerical variables where appropriate, and divide data according to the operational setting. Time-dependent problems generally require temporal rather than random splits. The output layer and loss must then be selected from the target type.

Architecture search should be controlled rather than unlimited. Begin with a modest number of layers and units, then vary width, depth, activation, dropout, weight decay, and learning rate using validation data or cross-validation. Early stopping can terminate training when validation performance ceases to improve. The selected model should be retrained according to a documented protocol and evaluated once on an untouched test set.

Evaluation must go beyond accuracy when class imbalance or unequal error costs exist. Classification may require precision, recall, F1 score, area under the precision-recall curve, calibration, subgroup performance, and threshold analysis. Regression may require mean absolute error, root mean squared error, error distributions, and performance across relevant segments. A strong average score does not guarantee reliable decisions for every population.

Illustrative Case: An MLP for Loan-Default Screening

Background

Consider a fictional regional lender, Meridian Cooperative Bank, seeking to improve its early screening of small-business loan applications. The bank has structured records containing revenue, debt-service ratio, business age, repayment history, industry category, transaction volatility, and the requested loan amount. The target is whether a loan becomes seriously delinquent within twelve months.

Situation and problem

The existing rule system is transparent but uses fixed thresholds that do not capture interactions. For example, a moderate debt-service ratio may be acceptable for a stable firm but risky for a firm with volatile cash flow. The bank wants a model that ranks applications for additional review, not one that automatically rejects applicants.

Analysis and approach

The data science team first defines the prediction point and excludes variables unavailable at that time. It uses a time-based split: earlier applications for training, a later period for validation, and the most recent period for testing. This design better reflects deployment and reduces temporal leakage. Numerical variables are standardized using training data, and categorical variables are encoded through a documented preprocessing pipeline.

The baseline is logistic regression. The proposed MLP contains an input layer, two hidden layers using ReLU activations, and a sigmoid output. He-style initialization is used for hidden layers. Because defaults are relatively infrequent, the team evaluates precision-recall performance, calibration, recall at a review-capacity threshold, and subgroup error rates rather than accuracy alone. Class weighting is considered, but its effect on probability calibration is checked.

The network is regularized with weight decay and early stopping. Hyperparameters are selected on the validation period, while the test period remains untouched. Explanation reports combine global sensitivity analyses, example-level reason codes derived from controlled feature perturbations, and the original policy rules. The MLP is not allowed to use protected attributes for prediction, but those attributes are retained in a restricted evaluation process to test disparate outcomes.

Outcome

In this illustrative case, the MLP improves ranking performance over the rule system and produces more useful review prioritization than the baseline. However, the bank does not deploy it as an autonomous approval or rejection mechanism. Applicants near the decision threshold receive human review, probability calibration is monitored, and performance is re-evaluated as economic conditions change.

Learning points

The case demonstrates that architecture is only one component of a reliable system. Temporal validation, leakage prevention, suitable metrics, calibration, human oversight, and subgroup analysis are as important as selecting ReLU or choosing the number of hidden units. A technically stronger predictor may still be inappropriate if its decisions cannot be governed or if its errors impose unequal burdens.

Failure Modes, Risks, and Responsible Use

Common implementation mistakes include omitting feature scaling, using an incompatible output activation and loss, initializing all weights identically, counting parameters incorrectly, and tuning on the test set. Another frequent error is assuming that a lower training loss indicates a better deployed model. Overfitting occurs when the network captures idiosyncrasies of the training sample; underfitting occurs when capacity or training is insufficient.

Data leakage can enter through preprocessing, duplicate records, future variables, target-derived fields, or inappropriate random splitting. Distribution shift may arise when customer behavior, sensors, regulations, or economic conditions change. Neural networks can also be poorly calibrated: a score of 0.8 need not correspond to an 80 percent event frequency unless calibration has been assessed.

For high-impact uses, fairness cannot be inferred from the absence of sensitive attributes. Correlated variables may reproduce protected characteristics, and optimization for aggregate performance may conceal subgroup failures. Organizations should document data provenance, intended use, limitations, monitoring responsibilities, escalation procedures, and model-change controls. Privacy-preserving data management, access control, retention limits, and security against adversarial or data-poisoning attacks are also relevant.

Interpretability should be treated as a governance requirement rather than a promise that every hidden unit has a simple human meaning. Where explanations influence decisions, they should be tested for stability and usefulness to affected users. Human oversight must be substantive: reviewers need authority, information, training, and a process for challenging model outputs.

Emerging Directions

Current development increasingly focuses on efficient and reliable representation learning rather than simply increasing network size. Automated architecture and hyperparameter search can explore MLP designs, but its search space and evaluation protocol must be governed to prevent validation overfitting. Self-supervised pretraining can learn representations from unlabeled tabular or multimodal data before supervised fine-tuning, although the value depends strongly on the structure of the domain.

Other directions include uncertainty estimation, conformal prediction, calibrated decision support, structured tabular architectures, and hybrid systems that combine neural representations with rules or domain constraints. Responsible deployment is also becoming more central: monitoring drift, auditing subgroup performance, documenting model cards, and enabling reproducible training are now part of professional neural-network engineering.

Best Practices for Feedforward Network Design

1. Define the target, decision context, and error costs before selecting an architecture.

2. Establish a transparent baseline and compare the MLP against it.

3. Use leakage-resistant preprocessing and scale features when appropriate.

4. Match the output activation and loss to the target distribution.

5. Initialize weights according to activation and layer dimensions; verify gradients and activations.

6. Start with a modest architecture and control complexity through validation, regularization, and early stopping.

7. Evaluate calibration, subgroup behavior, robustness, and operational thresholds—not only aggregate accuracy.

8. Preserve an untouched test set and record seeds, data versions, preprocessing, and hyperparameters.

9. Treat learned representations as task-dependent and inspect them with multiple complementary methods.

10. Deploy with monitoring, human governance, security controls, and a documented rollback process.

Chapter Summary

Feedforward networks transform inputs through successive affine mappings and nonlinear activations. The perceptron provides the basic linear decision unit, while the MLP composes multiple units to represent nonlinear functions. Activation functions determine both expressive behavior and gradient flow; ReLU variants are common in hidden layers, while sigmoid, softmax, or linear outputs are selected according to the target. Forward propagation evaluates the network and produces the intermediate values needed for learning. Parameter initialization breaks symmetry and helps preserve usable signal and gradients. Width, depth, bottlenecks, and skip connections shape the representations learned by the network. Effective application requires careful preprocessing, leakage-resistant validation, appropriate metrics, regularization, calibration, governance, and monitoring. The quality of a feedforward system therefore depends on the interaction of architecture, data, optimization, evaluation, and organizational responsibility.

Key Terms

Perceptron: A computational unit that applies a weighted sum, bias, and threshold or activation to its inputs.

Multilayer perceptron: A feedforward network containing one or more hidden layers of interconnected units.

Affine transformation: A linear mapping followed by addition of a bias, expressed as \(W\mathbf{x}+b\).

Activation function: A function applied to a unit’s pre-activation to introduce a transformation, usually nonlinearity.

ReLU: The activation \(\max(0,z)\), widely used in hidden layers.

Sigmoid: An activation mapping real values to the interval \((0,1)\).

Softmax: A function converting a vector of logits into nonnegative values that sum to one.

Forward propagation: The computation of predictions and intermediate activations from input to output.

Hidden layer: A network layer between the input and output that learns intermediate features.

Representation learning: Learning useful internal feature transformations directly from data.

Topology: The pattern of connections among units and layers in a network.

Width: The number of computational units in a network layer.

Depth: The number of transformation layers in a network.

Parameter initialization: The procedure for setting weights and biases before training.

He initialization: A variance-scaled initialization commonly used with ReLU-like activations.

Xavier initialization: A variance-scaled initialization associated with linear or tanh-like activations.

Saturation: A region where an activation derivative becomes very small, weakening gradients.

Bottleneck: A deliberately narrow representation layer that restricts information capacity.

Calibration: The agreement between predicted probabilities and observed event frequencies.

Data leakage: Unintended use of information during training that would not be available at prediction time.

Review Questions

Short Answer Questions

1. Why can a single perceptron not solve the XOR problem?

2. What role does a bias play in an affine neural-network layer?

3. Distinguish sigmoid and softmax output layers.

4. Why is identical initialization of all weights problematic?

5. What is meant by a bottleneck representation?

Long Answer Questions

1. Explain the architecture and mathematical operation of a multilayer perceptron, including forward propagation through two hidden layers.

2. Compare sigmoid, tanh, ReLU, and Leaky ReLU with respect to range, saturation, sparsity, and gradient behavior.

3. Discuss how depth, width, and topology influence representation learning and optimization.

4. Explain He and Xavier initialization and the problems caused by poorly scaled parameters.

5. Develop a complete workflow for designing, validating, and deploying an MLP for a tabular classification problem.

Analytical / Application Questions

1. Design the output layer, activation, and loss function for each of the following: a three-class exclusive classification task, a multilabel diagnosis task, and a nonnegative demand-forecasting task. Justify each choice.

2. An MLP has excellent training accuracy but poor performance on a later time period. Analyze possible causes and propose a validation and monitoring strategy.

3. Using the Meridian Cooperative Bank case, explain how leakage, calibration, subgroup evaluation, and human review should influence the deployment decision.

22. Multiple Choice Questions with Answers

Question 1

Which statement best explains why a multilayer perceptron needs nonlinear hidden activations?

A. They reduce the number of trainable parameters.

B. They allow compositions of layers to represent nonlinear functions.

C. They guarantee that the model will not overfit.

D. They remove the need for a loss function.

Correct Answer: B. They allow compositions of layers to represent nonlinear functions.

Question 2

For a mutually exclusive three-class classification problem, which output design is most appropriate?

A. Three independent linear outputs with mean squared error only

B. One sigmoid output with binary cross-entropy

C. Three sigmoid outputs with multilabel interpretation

D. Three logits followed by softmax with categorical cross-entropy

Correct Answer: D. Three logits followed by softmax with categorical cross-entropy

Question 3

What is the main reason for using random weight initialization rather than setting every weight in a layer to the same value?

A. To break symmetry so units can learn different features

B. To ensure every activation is positive

C. To eliminate the need for feature scaling

D. To make the model deterministic across runs

Correct Answer: A. To break symmetry so units can learn different features

Question 4

A layer uses ReLU activations and its weights are initialized with variance approximately proportional to 2 divided by the number of incoming connections. Which initialization is being used?

A. Zero initialization

B. Xavier initialization

C. He initialization

D. One-hot initialization

Correct Answer: C. He initialization

Question 5

Which output activation is generally suitable for unconstrained real-valued regression?

A. Linear

B. Softmax

C. Sigmoid

D. Hard threshold

Correct Answer: A. Linear

Question 6

A model is trained after standardizing features using statistics calculated from the entire dataset, including the test set. What problem does this create?

A. Underparameterization

B. Data leakage

C. Symmetry breaking

D. Class exclusivity

Correct Answer: B. Data leakage

Question 7

What is a likely consequence of using very large initial weights in a deep network?

A. All units necessarily become identical

B. Activations and gradients may explode or saturate

C. The model becomes a linear classifier

D. The output is automatically calibrated

Correct Answer: B. Activations and gradients may explode or saturate

Question 8

Which statement most accurately describes a learned hidden representation?

A. It is always a universally meaningful feature description independent of the task.

B. It is a transformation shaped by the objective and data used for training.

C. It consists only of the original input variables in a new order.

D. It cannot encode sensitive or spurious correlations.

Correct Answer: B. It is a transformation shaped by the objective and data used for training.

Question 9

A network is used for multilabel classification in which several labels can be true simultaneously. Which output design is appropriate?

A. Independent sigmoid outputs

B. A single softmax output

C. A hard threshold before the hidden layers

D. A single linear output for all labels

Correct Answer: A. Independent sigmoid outputs

Question 10

In the illustrative loan-screening case, why was a time-based split preferred to a random split?

A. It always produces a larger training set.

B. It removes the need for a validation set.

C. It better reflects deployment over time and reduces temporal leakage.

D. It guarantees equal performance across demographic groups.

Correct Answer: C. It better reflects deployment over time and reduces temporal leakage.