Neural Networks and Representation Learning

Introduction and Context

Neural networks are computational models that learn relationships between inputs and outputs by composing many simple transformations. Their importance in artificial intelligence arises from two capabilities: function approximation, the ability to model complex nonlinear relationships, and representation learning, the ability to discover useful features directly from data.

Earlier machine-learning systems often depended on manually designed features. For example, an image classifier might use edge detectors, color histograms, or texture descriptors created by an engineer. A neural network can learn successive representations: early layers may detect simple patterns, intermediate layers may combine them into shapes, and later layers may represent objects or semantic categories. TensorFlow and PyTorch provide the computational systems needed to define, train, evaluate, and deploy these models.

A neural network is not automatically intelligent. Its performance depends on data quality, architecture, objective function, optimization, regularization, and evaluation. Understanding these foundations is therefore essential for both responsible use and effective implementation.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain perceptrons and multilayer neural networks.

2. Distinguish linear, nonlinear, output, and loss functions.

3. Describe forward propagation and backpropagation mathematically.

4. Explain automatic differentiation in TensorFlow and PyTorch.

5. Select suitable initialization and optimization methods.

6. Analyze overfitting, regularization, and generalization.

7. Implement and evaluate a basic neural network responsibly.

1. The Perceptron: The Basic Computational Unit

A perceptron receives an input vector \(\mathbf{x}=(x_1,\ldots,x_d)\), computes a weighted sum, adds a bias, and applies an activation function:

\[

z=\mathbf{w}^{T}\mathbf{x}+b, \qquad \hat{y}=\phi(z).

\]

Here, \(\mathbf{w}\) is a vector of learnable weights, \(b\) is a learnable bias, \(\phi\) is an activation function, and \(\hat y\) is the prediction. The bias allows the decision boundary to shift rather than being forced through the origin.

The original binary perceptron used a step function, producing one class when \(z\geq 0\) and another otherwise. It can represent linearly separable patterns, such as a logical AND, but cannot represent XOR with a single unit. The perceptron learning rule can converge for linearly separable data under suitable conditions, but it does not solve general nonlinear problems.

A perceptron is therefore best understood as a linear decision unit, not as a complete modern neural network. Nonlinear representation learning requires multiple layers and differentiable or suitably trainable activations.

2. From Single Units to Multilayer Networks

A feedforward neural network, or multilayer perceptron (MLP), consists of an input layer, one or more hidden layers, and an output layer. For layer \(l\), the usual equations are

\[

\mathbf{z}^{(l)}=\mathbf{W}^{(l)}\mathbf{a}^{(l-1)}+\mathbf{b}^{(l)},

\qquad

\mathbf{a}^{(l)}=\phi^{(l)}(\mathbf{z}^{(l)}),

\]

where \(\mathbf{a}^{(0)}=\mathbf{x}\). The matrix \(\mathbf W^{(l)}\) maps the previous layer to the current layer, and vectors are commonly represented as column vectors. Dimensions must match: if layer \(l-1\) has \(n_{l-1}\) units and layer \(l\) has \(n_l\) units, then \(\mathbf W^{(l)}\) has shape \(n_l\times n_{l-1}\), and \(\mathbf b^{(l)}\) has shape \(n_l\).

A composition of affine transformations without nonlinear activations remains an affine transformation. Consequently, hidden-layer nonlinearities are essential. They enable the network to form curved decision boundaries and learn hierarchical representations.

Representation Learning

A representation is a transformation of raw data into features useful for a task. In an image network, early representations may encode local intensity changes, while deeper representations combine these patterns. In a language model, representations may encode word usage and contextual relationships. The features are not necessarily human-interpretable, but they can support prediction.

The universal approximation theorem states, informally, that a sufficiently large feedforward network with an appropriate non-polynomial activation can approximate broad classes of continuous functions on compact domains. This is an existence result, not a guarantee that training will find the function, that the required network will be small, or that the learned model will generalize well.

3. Activation Functions

Activation functions determine how signals are transformed and how gradients flow.

Sigmoid

\[

\sigma(z)=\frac{1}{1+e^{-z}}.

\]

Its output lies in \((0,1)\), making it useful for a binary probability output when paired with an appropriate loss. Its derivative is \(\sigma(z)(1-\sigma(z))\). For large positive or negative values, the derivative becomes small, creating a vanishing-gradient problem in hidden layers.

Hyperbolic Tangent

\[

\tanh(z)=\frac{e^z-e^{-z}}{e^z+e^{-z}}.

\]

It produces values in \((-1,1)\) and is zero-centered, but it can also saturate and produce small gradients.

ReLU

\[

\operatorname{ReLU}(z)=\max(0,z).

\]

ReLU is computationally simple and generally provides stronger gradients for positive inputs. A unit that remains negative for all training examples may stop updating, a condition called the dying-ReLU problem.

Leaky ReLU and Related Functions

A leaky ReLU uses a small positive slope for negative values:

\[

f(z)=\begin{cases}z,&z\geq0\\\alpha z,&z<0,

\end{cases}

\]

where \(\alpha\) is small and positive. GELU and SiLU are smoother alternatives frequently used in modern architectures. The best choice depends on architecture, data, initialization, and empirical validation.

Output Activations

For mutually exclusive classes, softmax converts logits \(\mathbf z\) into probabilities:

\[

p_i=\frac{e^{z_i}}{\sum_{j=1}^{K}e^{z_j}}.

\]

The probabilities are nonnegative and sum to one. In implementation, numerically stable combined logits-loss functions should generally be used rather than manually computing exponentials. For independent binary labels, sigmoid outputs are appropriate. For regression, a linear output is common unless the target has a constrained range.

4. Loss Functions and Learning Objectives

A loss function measures disagreement between predictions and targets. Training minimizes an empirical objective, often the mean loss over a mini-batch:

\[

J(\theta)=\frac{1}{m}\sum_{i=1}^{m}\ell(f_\theta(\mathbf{x}_i),y_i),

\]

where \(m\) is the batch size, \(f_\theta\) is the network with parameters \(\theta\), and \(\ell\) is the per-example loss.

For regression, mean squared error is

\[

\operatorname{MSE}=\frac{1}{m}\sum_{i=1}^{m}(\hat y_i-y_i)^2.

\]

It penalizes large errors strongly and is appropriate when squared-error assumptions are reasonable. Mean absolute error, \(\frac1m\sum_i|\hat y_i-y_i|\), is less sensitive to extreme errors but is not differentiable at zero; modern optimizers can still handle it using subgradients or smooth alternatives.

For binary classification, binary cross-entropy for probability \(p\) and target \(y\in\{0,1\}\) is

\[

\ell=-[y\log p+(1-y)\log(1-p)].

\]

For multiclass classification with one correct class, cross-entropy is \(-\log p_y\), where \(p_y\) is the predicted probability assigned to the true class. Cross-entropy encourages calibrated probability mass on the correct class, although calibration is not guaranteed.

The loss is a modeling choice. It should reflect the prediction task, target structure, and consequences of errors. Class imbalance may require class weighting, resampling, focal loss, or suitable evaluation metrics, but these choices must be validated rather than applied automatically.

5. Forward Propagation

Forward propagation evaluates the network from input to output. Consider a network with one hidden layer:

\[

\mathbf z^{(1)}=\mathbf W^{(1)}\mathbf x+\mathbf b^{(1)},

\quad

\mathbf a^{(1)}=\phi(\mathbf z^{(1)}),

\quad

\mathbf z^{(2)}=\mathbf W^{(2)}\mathbf a^{(1)}+\mathbf b^{(2)}.

\]

For binary classification, the final probability is \(\hat y=\sigma(z^{(2)})\). During inference, the network performs this sequence without changing parameters. During training, the output is compared with the target, and gradients are computed.

Worked Example: One Neuron

Suppose \(x_1=2\), \(x_2=-1\), \(w_1=0.4\), \(w_2=-0.6\), and \(b=0.1\). Then

\[

z=(0.4)(2)+(-0.6)(-1)+0.1=1.5.

\]

With a sigmoid output,

\[

\hat y=\frac{1}{1+e^{-1.5}}\approx0.818.

\]

Thus the neuron assigns approximately 81.8% probability to the positive class. This value is not a guarantee that the example belongs to that class; it is the model’s estimated probability under its learned parameters.

6. Backpropagation and Automatic Differentiation

Backpropagation efficiently computes derivatives of the loss with respect to all parameters by applying the chain rule from the output toward the input. It is not itself an optimization algorithm; it supplies gradients to an optimizer.

For a scalar output with squared loss

\[

L=\frac12(\hat y-y)^2,

\]

the factor \(1/2\) is convenient because it cancels a factor of 2 during differentiation. For a simple neuron \(\hat y=\phi(z)\), \(z=\mathbf w^T\mathbf x+b\),

\[

\frac{\partial L}{\partial w_j}=(\hat y-y)\phi'(z)x_j,

\qquad

\frac{\partial L}{\partial b}=(\hat y-y)\phi'(z).

\]

For a multilayer network, define \(\boldsymbol\delta^{(l)}=\partial L/\partial\mathbf z^{(l)}\). At the output, the exact expression depends on the output activation and loss. For a hidden layer,

\[

\boldsymbol\delta^{(l)}=

\left((\mathbf W^{(l+1)})^T\boldsymbol\delta^{(l+1)}\right)\odot \phi'(\mathbf z^{(l)}),

\]

where \(\odot\) denotes elementwise multiplication. The parameter gradients are

\[

\frac{\partial L}{\partial\mathbf W^{(l)}}=

\boldsymbol\delta^{(l)}(\mathbf a^{(l-1)})^T,

\qquad

\frac{\partial L}{\partial\mathbf b^{(l)}}=\boldsymbol\delta^{(l)}.

\]

These formulas assume the stated column-vector convention and one example; batch implementations aggregate or average corresponding gradients.

Automatic Differentiation

TensorFlow and PyTorch construct computational graphs from tensor operations. Automatic differentiation applies the chain rule programmatically. It is more reliable than manually deriving every gradient, but it does not make an incorrect model correct. Common implementation errors include wrong tensor shapes, applying an activation twice, using a loss incompatible with the output layer, and accidentally disabling gradient tracking.

A typical PyTorch training pattern is:

```python

for X, y in loader:

optimizer.zero_grad()

logits = model(X)

loss = loss_fn(logits, y)

loss.backward()

optimizer.step()

```

In TensorFlow/Keras, a compiled model commonly uses `model.fit`; custom training uses `tf.GradientTape` to record operations and calculate gradients. In both frameworks, parameters are updated only after gradients are computed and passed to an optimizer.

7. Initialization and Gradient Flow

Initialization strongly affects optimization. If all hidden units begin with identical weights, they receive identical gradients and remain equivalent; this is the symmetry problem. Random initialization breaks symmetry.

For a layer with \(n_{in}\) inputs and \(n_{out}\) outputs, Xavier/Glorot initialization aims to maintain a suitable variance across layers and is commonly used with sigmoid or tanh-like activations. He initialization uses a variance approximately \(2/n_{in}\) for ReLU-family activations. These are variance-based guidelines, not universal guarantees.

Poor initialization can cause vanishing gradients, in which gradients become extremely small, or exploding gradients, in which they become excessively large. Normalization methods, residual connections, suitable activations, gradient clipping, and careful learning rates can help, but each introduces design trade-offs.

8. Optimization

The basic gradient-descent update is

\[

\theta_{t+1}=\theta_t-\eta\nabla_\theta J(\theta_t),

\]

where \(\eta>0\) is the learning rate. Full-batch gradient descent uses all training examples; stochastic gradient descent uses one example; mini-batch SGD uses a subset and is standard in practice.

Momentum accumulates a moving direction to reduce oscillation. Adaptive methods such as Adam maintain estimates of first and second moments of gradients. Adam often provides rapid early progress, whereas SGD with momentum can sometimes provide strong final generalization. No optimizer is universally best.

Learning-rate schedules, warm-up, decay, and early stopping can improve training. Batch size affects both computational efficiency and gradient noise. Training loss, validation loss, learning curves, and task-specific metrics should be inspected together rather than relying on a single final score.

9. Regularization and Generalization

A model generalizes when it performs well on unseen examples drawn from the relevant deployment distribution. A large gap between training and validation performance is evidence of overfitting, although distribution shift can produce similar symptoms.

Common Techniques

• L2 regularization (weight decay): adds a penalty proportional to \(\|\mathbf w\|_2^2\), discouraging excessively large weights. The precise relationship between classical L2 penalties and optimizer weight decay depends on the optimizer.
• L1 regularization: adds \(\lambda\|\mathbf w\|_1\), often encouraging sparse parameters.
• Dropout: randomly sets units to zero during training with a specified probability and rescales activations according to the framework’s convention. It is disabled during evaluation.
• Early stopping: ends training when validation performance ceases to improve, using a validation set not used for parameter fitting.
• Data augmentation: creates plausible variations, such as image crops or flips, while preserving labels.
• Data and model design: reducing unnecessary capacity, collecting representative data, and preventing leakage are often more effective than adding penalties alone.

A validation set supports model selection; the test set should be used sparingly for final unbiased assessment. Hyperparameter tuning against the test set contaminates the estimate of deployment performance.

10. An Illustrative Applied Case: Defect Detection in Manufacturing

Background. A factory wants to identify defective components from camera images. Defects are uncommon, and false negatives may be more costly than false positives.

Problem. A network achieves 99% accuracy, but only because 99% of components are normal. Its recall for defects is poor, making accuracy misleading.

Analysis. Engineers first separate training, validation, and test images by production batch or time period to reduce near-duplicate leakage. They inspect class frequencies, label quality, lighting variation, and camera changes. A confusion matrix, precision, recall, PR-AUC, and cost-weighted analysis are more informative than accuracy alone.

Approach. Images are normalized using training-set statistics. A convolutional model is initialized from a suitable pretrained model, with augmentation that reflects realistic orientation and lighting. A weighted classification loss is considered, and the decision threshold is selected on validation data according to the cost of missed defects. The test set is evaluated only after the model and threshold are finalized.

Outcome. The resulting system is not judged merely by its training loss. It is monitored for changes in camera conditions, defect types, and subgroup performance. Human inspectors review uncertain cases, and predictions are logged for audit and retraining.

Learning points. The case demonstrates that representation learning, optimization, evaluation, threshold selection, and deployment governance are inseparable. A technically sophisticated network can still fail if labels leak, metrics are inappropriate, or the deployment distribution changes.

11. Professional, Ethical, and Governance Considerations

Neural networks can encode bias in data and amplify it through optimization. Practitioners should document data sources, consent and permissible use, labeling procedures, known limitations, and subgroup performance. Sensitive applications require privacy protection, access controls, audit trails, and human oversight.

A model’s confidence is not proof of correctness. Calibration, uncertainty analysis, robustness testing, and monitoring are important where errors affect safety, employment, health, finance, or access to services. Reproducibility requires recording random seeds where possible, software versions, preprocessing, architecture, hyperparameters, and dataset versions. Governance frameworks should assign responsibility for approval, deployment, incident response, and retirement.

12. Contemporary Developments and Best Practices

Modern representation learning increasingly uses pretrained models, transfer learning, self-supervised objectives, attention mechanisms, and multimodal architectures. These methods can reduce task-specific labeling requirements but may introduce large computational costs, inherited biases, licensing constraints, and opaque data provenance.

Recommended practice is to begin with a simple baseline, establish a reliable data split, choose a loss aligned with the task, normalize inputs, use appropriate initialization, monitor training and validation curves, and conduct error analysis. Keep the test set isolated, compare against non-neural baselines, and evaluate operational consequences. Framework convenience should not replace mathematical and experimental understanding.

Chapter Summary

A perceptron computes an affine transformation followed by an activation. Multilayer networks compose such transformations, and nonlinear activations allow them to learn representations and nonlinear decision boundaries. Forward propagation produces predictions; backpropagation applies the chain rule to calculate parameter gradients; automatic differentiation implements this process in TensorFlow and PyTorch. Loss functions express the training objective, while gradient-based optimizers update parameters. Initialization affects symmetry and gradient flow. Regularization, validation discipline, augmentation, early stopping, and appropriate capacity help control overfitting and improve generalization. Successful professional use also requires sound data splitting, meaningful metrics, monitoring, documentation, fairness analysis, privacy protection, and human accountability.

Key Terms

Perceptron: A weighted-sum computational unit with an activation function.

Weight: A learnable coefficient controlling an input connection.

Bias: A learnable offset added before activation.

Activation function: A function that introduces a transformation, often nonlinearity.

Multilayer perceptron: A feedforward network containing one or more hidden layers.

Logit: An unnormalized output score, often produced before softmax or sigmoid.

Softmax: A function converting multiple logits into probabilities summing to one.

Loss function: A quantity measuring prediction error for training.

Forward propagation: Computation of network outputs from inputs.

Backpropagation: Efficient reverse application of the chain rule for gradients.

Automatic differentiation: Programmatic calculation of derivatives through recorded operations.

Gradient: The vector of partial derivatives of an objective with respect to parameters.

Epoch: One complete pass through the training dataset.

Mini-batch: A subset of examples used for one parameter update.

Initialization: The procedure for choosing starting parameter values.

Vanishing gradient: A gradient that becomes too small for effective learning.

Regularization: A method that discourages undesirable model complexity or behavior.

Overfitting: Strong training performance accompanied by poor unseen-data performance.

Generalization: Performance on relevant data not used for fitting.

Data leakage: Unintended use of information that would not be available at deployment.

Review Questions

Short Answer Questions

1. Why is a nonlinear activation necessary in a multilayer network?

2. What roles do weights and biases play in a neuron?

3. Distinguish forward propagation from backpropagation.

4. Why can accuracy be misleading for highly imbalanced classification?

5. What is the purpose of a validation set?

Long Answer Questions

1. Explain the architecture and operation of a multilayer perceptron, including the dimensions of its weight matrices.

2. Compare sigmoid, tanh, ReLU, and softmax, including appropriate uses and limitations.

3. Describe how automatic differentiation and an optimizer work together during training in PyTorch or TensorFlow.

4. Explain vanishing gradients, initialization strategies, and methods for improving gradient flow.

5. Discuss how regularization and evaluation protocols influence generalization.

Analytical / Application Questions

1. Design an evaluation plan for a neural classifier used in a safety-critical manufacturing process. Justify the data split, metrics, threshold, and monitoring strategy.

2. A model has low training loss but much higher validation loss. Analyze at least four possible causes and propose a sequence of diagnostic interventions.

3. For a binary classifier with rare positive cases, compare class weighting, data augmentation, threshold adjustment, and collection of additional data. Explain when each may help and what risks it introduces.

Multiple Choice Questions with Answers

Question 1

What limitation does a single perceptron have?

A. It cannot compute a weighted sum

B. It cannot represent nonlinearly separable patterns such as XOR

C. It cannot use a bias term

D. It cannot be trained with labeled data

Correct Answer: B. It cannot represent nonlinearly separable patterns such as XOR

Question 2

Which activation is commonly used to convert mutually exclusive class logits into probabilities summing to one?

A. ReLU

B. Tanh

C. Softmax

D. Leaky ReLU

Correct Answer: C. Softmax

Question 3

What is the primary purpose of backpropagation?

A. To initialize weights randomly

B. To compute gradients of the loss with respect to parameters

C. To divide data into training and test sets

D. To convert labels into images

Correct Answer: B. To compute gradients of the loss with respect to parameters

Question 4

If a layer has 12 inputs and 5 output units, what is the shape of its weight matrix under the column-vector convention used in the chapter?

A. 12 × 5

B. 5 × 12

C. 12 × 12

D. 5 × 5

Correct Answer: B. 5 × 12

Question 5

Which method is specifically intended to reduce symmetry among hidden units at initialization?

A. Giving every unit identical weights

B. Randomly initializing weights

C. Removing all biases

D. Setting all activations to zero permanently

Correct Answer: B. Randomly initializing weights

Question 6

Which loss is commonly appropriate for regression when squared error is the chosen objective?

A. Mean squared error

B. Multiclass cross-entropy only

C. Softmax loss with class indices

D. Hinge thresholding without targets

Correct Answer: A. Mean squared error

Question 7

What is a major sign of overfitting?

A. Similar strong performance on training and validation data

B. High training performance but substantially poorer validation performance

C. A model having any hidden layer

D. Using mini-batches

Correct Answer: B. High training performance but substantially poorer validation performance

Question 8

Why should the test set generally be kept isolated until final evaluation?

A. To prevent test information from influencing model selection

B. Because test data cannot contain labels

C. Because optimizers cannot process test examples

D. To guarantee perfect accuracy

Correct Answer: A. To prevent test information from influencing model selection

Question 9

For a binary classifier with rare positive cases, which metric is often more informative than accuracy alone?

A. The number of model parameters only

B. Recall and precision, considered with the confusion matrix

C. The input image width only

D. The training batch size only

Correct Answer: B. Recall and precision, considered with the confusion matrix

Question 10

In the chapter’s one-neuron example, what is the pre-activation value z?

A. 0.1

B. 0.5

C. 1.5

D. 2.1

Correct Answer: C. 1.5