Neural Networks and Learning Processes

Introduction

Neural networks are computational models inspired loosely by biological nervous systems. In deep learning, they learn a mapping from inputs to outputs by adjusting numerical parameters called weights. This chapter develops the foundations required to understand how neural networks represent patterns, calculate predictions, and improve through training. These ideas are central to applications such as text classification, sentiment analysis, speech recognition, machine translation, and language modelling.

A neural network does not “understand” data in the human sense. It identifies statistical regularities in examples. Its usefulness therefore depends on the quality of the data, the suitability of the architecture, the objective function, and the methods used to control error and overfitting.

Learning Objectives

By the end of this chapter, you should be able to:

• explain the structure and operation of a perceptron;
• distinguish single-layer and multilayer neural networks;
• select suitable activation functions;
• describe forward propagation and calculate a simple prediction;
• explain backpropagation and gradient-based learning;
• compare parameter initialization methods;
• explain regularization, normalization, and dropout;
• identify important hyperparameters;
• diagnose and control overfitting in practical projects.

1. The Perceptron: A Basic Computational Unit

A perceptron receives input features, multiplies each by a weight, adds a bias, and applies an activation function. For inputs x₁, x₂, …, xₙ and corresponding weights w₁, w₂, …, wₙ, the weighted sum is:

Here, z is the pre-activation value, wᵢ is the weight for feature xᵢ, and b is the bias. The bias allows the decision boundary to shift rather than being forced through the origin. A classical binary perceptron uses a step function:

The output ŷ is a predicted class. The perceptron can learn linearly separable problems, meaning that a straight line, plane, or hyperplane can separate the classes. It cannot represent the XOR relationship with a single layer because XOR is not linearly separable.

A perceptron learning rule changes the weights when an example is misclassified. A simplified update is:

where y is the target class, η is the learning rate, and ŷ is the prediction. This rule is historically important, but modern neural networks usually use differentiable activation functions and gradient-based optimization.

2. From Perceptrons to Multilayer Networks

A multilayer perceptron (MLP) contains an input layer, one or more hidden layers, and an output layer. Each layer transforms its input and passes the result to the next layer. A network with several hidden layers is commonly called a deep neural network.

The key advantage of hidden layers is representation learning. Early layers may learn simple combinations of features, while later layers combine these representations into more abstract patterns. In natural language processing, representations may encode word identity, local word combinations, syntactic information, or semantic relationships, depending on the architecture and training objective.

For layer l, the general transformation is:

Here, a⁽ˡ⁻¹⁾ is the previous layer’s activation, W⁽ˡ⁾ is the weight matrix, b⁽ˡ⁾ is the bias vector, z⁽ˡ⁾ is the pre-activation vector, and f⁽ˡ⁾ is an element-wise activation function. The input can be represented as a⁽⁰⁾ = x.

A network is called “feed-forward” when information moves from input to output without recurrent cycles. Many introductory MLPs are feed-forward networks. Other architectures, including recurrent networks and transformers, use additional structures for sequential or contextual processing.

3. Activation Functions

Without nonlinear activation functions, stacking linear layers still produces only a linear transformation. Nonlinearity enables a network to approximate complex functions.

3.1 Sigmoid

The sigmoid function is:

Its output lies between 0 and 1, making it useful for a binary probability output. Its derivative is:

However, sigmoid units can suffer from vanishing gradients when z is very positive or very negative. They are therefore less common in hidden layers of modern deep networks.

3.2 Hyperbolic tangent

The output lies between −1 and 1. Like sigmoid, tanh can saturate and produce small gradients, but its centred output may be advantageous in some settings.

3.3 ReLU

The rectified linear unit is:

Its derivative is 1 for positive z and 0 for negative z, apart from the undefined point at zero, which is assigned a practical convention. ReLU is computationally simple and often trains more effectively than sigmoid or tanh. A limitation is the “dead ReLU” problem: a unit that remains on the negative side may receive zero gradient and stop learning.

3.4 Variants

Leaky ReLU assigns a small non-zero slope to negative inputs:

Here α is a small positive constant, such as 0.01. Other variants include GELU, ELU, and the smooth function used in many transformer models, but the appropriate choice depends on architecture and empirical evaluation.

3.5 Softmax

For a K-class output, softmax converts logits z₁, …, z_K into a probability distribution:

Each pᵢ lies between 0 and 1 and the probabilities sum to 1. Softmax is normally paired with categorical cross-entropy for single-label multiclass classification. For numerical stability, implementations usually subtract the largest logit before exponentiation; this does not change the resulting probabilities.

4. Forward Propagation

Forward propagation is the process of computing a prediction from an input. At each layer, the network performs a matrix multiplication, adds a bias, and applies an activation function.

Worked Example: One Hidden-Layer Network

Suppose a network receives x = [1, 2]. It has two hidden units with:

Using ReLU, the hidden pre-activations are:

Therefore:

Let the output layer have weights [0.2, 0.4] and bias −0.1. Its logit is:

For a binary output, applying sigmoid gives approximately:

Thus the network predicts a positive-class probability of approximately 0.75. This calculation illustrates that the output depends on every preceding transformation and parameter.

5. Loss Functions and the Learning Objective

A neural network learns by minimizing a loss function over training examples. For binary classification, binary cross-entropy is:

where y ∈ {0, 1} is the target and p is the predicted probability. The formula assumes a Bernoulli target and a prediction interpreted as a probability.

For K-class single-label classification, categorical cross-entropy is:

where yᵢ is 1 for the correct class and 0 otherwise when one-hot targets are used. For regression, mean squared error is commonly written:

where m is the number of examples, yᵢ is the observed value, and ŷᵢ is the prediction. The loss function must match the task and output interpretation.

6. Backpropagation and Gradient Descent

Backpropagation efficiently computes how much the loss changes with respect to every weight and bias. It applies the chain rule of calculus from the output layer backwards through the network.

For a parameter θ, gradient descent updates it as:

The gradient ∂L ÷ ∂θ indicates the direction of greatest increase in loss, so subtracting it reduces the loss for a sufficiently small learning rate η.

For a simple output neuron with z = wx + b, output p = σ(z), and binary cross-entropy loss, the derivative with respect to z simplifies to:

Consequently:

These expressions assume sigmoid output and binary cross-entropy. Other activation-loss combinations produce different derivatives.

In practice, training usually uses mini-batch gradient descent. A mini-batch is a subset of the training data. The average gradient over the batch is used to update parameters. Stochastic gradient descent uses one example at a time, while full-batch gradient descent uses all training examples per update.

Modern optimizers such as momentum, RMSProp, and Adam modify the basic update using running estimates of gradients or squared gradients. They can improve convergence, but they introduce additional hyperparameters and do not eliminate the need for validation and careful experimental design.

7. Parameter Initialization

Initialization determines the starting values of weights. Poor initialization can cause activations or gradients to become extremely large or extremely small. Initializing all weights to zero is unsuitable for hidden layers because neurons receive identical gradients and remain identical, a problem known as failure of symmetry breaking. Biases are often initialized to zero, although architecture-specific alternatives exist.

Two widely used variance-aware methods are:

• Glorot/Xavier initialization: suitable when the effects of the incoming and outgoing layers should be balanced. A common uniform version samples from a range based on 6 ÷ (n_in + n_out), where n_in and n_out are the numbers of input and output units.
• He initialization: designed especially for ReLU-like activations and commonly uses variance approximately 2 ÷ n_in.

These are distributional guidelines rather than guarantees. The exact implementation may use uniform or normal distributions, and the best choice depends on the activation function and architecture.

8. Regularization and Normalization

8.1 Weight regularization

L2 regularization adds a penalty proportional to the squared weights:

L1 regularization adds a penalty proportional to absolute weight values:

Here λ ≥ 0 controls penalty strength. L2 tends to shrink weights smoothly, whereas L1 can encourage exact zeros and therefore a sparse parameter vector. The penalty convention sometimes includes a factor of 1 ÷ 2 or averages over parameters; such conventions alter the gradient scaling but not the basic principle.

8.2 Dropout

During training, dropout randomly sets a proportion of activations to zero. With inverted dropout, surviving activations are scaled during training so that the expected activation remains approximately unchanged at inference. Dropout reduces reliance on particular units and can improve generalization, although excessive dropout may cause underfitting.

8.3 Early stopping

Early stopping monitors validation performance and stops training when it no longer improves, often after a patience period. It is an effective practical form of regularization, provided that the test set is not used repeatedly for stopping decisions.

8.4 Normalization

Normalization methods improve the scale and conditioning of learning. Input standardization commonly uses:

where μ is the training-set mean and σ is the training-set standard deviation. Statistics must be computed using the training split and then applied unchanged to validation and test data, preventing information leakage.

Batch normalization normalizes intermediate activations using mini-batch statistics during training and learned scale and shift parameters. Layer normalization instead normalizes across features within an individual example and is particularly common in transformer-based NLP systems. Normalization can stabilize optimization, but it does not replace good initialization, appropriate learning rates, or careful data preparation.

9. Hyperparameters and Training Design

Parameters are learned from data; hyperparameters are selected by the practitioner. Important hyperparameters include:

The learning rate is often the most influential optimization hyperparameter. If too large, training may diverge or oscillate; if too small, learning may be extremely slow or become trapped in an ineffective region. Learning-rate schedules reduce or otherwise modify the rate during training. Hyperparameters should be selected using a validation set or cross-validation where appropriate, while the test set should be reserved for final evaluation.

10. Overfitting and Its Control

Overfitting occurs when a model performs well on training data but poorly on unseen data. It usually reflects excessive effective capacity, insufficient data, noisy labels, data leakage, or an overly long training process. Underfitting occurs when the model cannot represent the underlying pattern or has not been trained sufficiently.

A useful diagnostic is to compare training and validation curves. If training loss continues to decrease while validation loss rises, overfitting is likely. If both losses remain high, the model may be underfitting or the optimization may be ineffective.

Control strategies include:

1. obtaining more representative training data;
2. reducing model size;
3. applying dropout or weight regularization;
4. using early stopping;
5. improving data quality and labels;
6. applying suitable augmentation or preprocessing;
7. tuning the learning rate and batch size;
8. avoiding leakage between training, validation, and test data.

In NLP, duplicate documents, near-identical texts, or author-specific signals can produce misleadingly high scores. Splits should reflect the intended deployment setting. For example, if a classifier will process future documents, a time-based split may be more realistic than a random split.

11. Applied Case: Sentiment Classification of Product Reviews

This is an illustrative case. A retailer wants to classify reviews as positive or negative to identify emerging service problems.

Background and problem

The available data contain 20,000 labelled reviews. Each review is converted into a numerical representation, such as a bag-of-words vector, a pretrained embedding, or a sequence representation. The objective is binary classification.

Analysis

A baseline logistic regression model is first trained. An MLP is then considered because sentiment may depend on nonlinear combinations of words, such as negation and intensification. The dataset is divided into training, validation, and test sets. Duplicate reviews are removed before splitting to reduce leakage.

Approach

The MLP contains an input representation, a hidden layer using ReLU, dropout, and a sigmoid output. Binary cross-entropy is used as the loss. Weights are initialized with a method suitable for the hidden activation. Input statistics, if required, are learned only from the training set. Training is monitored using validation loss and validation F1-score, especially if positive and negative classes are imbalanced.

If validation loss begins increasing while training loss continues decreasing, early stopping is applied. Class weights or resampling may be considered if one class is underrepresented. The final test set is evaluated only after model selection is complete.

Outcome and interpretation

Suppose the MLP improves recall for negative reviews but produces more false positives. This may be acceptable if identifying dissatisfied customers is more important than minimizing unnecessary alerts. Accuracy alone is therefore insufficient; precision, recall, F1-score, and a confusion matrix should be examined according to business costs.

Learning points

The case demonstrates that architecture is only one part of a reliable solution. Data splitting, leakage prevention, class balance, regularization, threshold selection, and operational objectives are equally important. A model should also be checked for systematic errors involving language variety, product category, or culturally specific expressions.

12. Limitations, Ethics, and Professional Practice

Neural networks can encode biases in training data and may perform unevenly across demographic groups, dialects, or domains. In NLP, a sentiment classifier may misinterpret sarcasm, code-switching, minority dialects, or culturally specific language. Evaluation should therefore include subgroup analysis where lawful and ethically justified.

Predictions should not be treated as unquestionable facts. Important applications require documentation of training data, intended use, known limitations, version history, and monitoring procedures. Personal or sensitive text must be processed according to applicable privacy and data-protection requirements. Data minimization, access control, retention limits, and secure storage are essential.

A further limitation is distribution shift: the relationship between inputs and targets may change after deployment. Monitoring should track changes in input patterns, error rates, class frequencies, and subgroup performance. Human review is appropriate when decisions have significant consequences.

13. Contemporary Developments

Current NLP systems often combine learned representations with attention mechanisms and transformer architectures. These models still rely on the same foundational processes: weighted transformations, nonlinear functions, loss minimization, gradient-based learning, initialization, normalization, and regularization. Large pretrained models introduce additional concerns, including computational cost, data provenance, hallucinated content, copyright, privacy, and efficient adaptation through fine-tuning or parameter-efficient methods.

Understanding basic neural-network learning remains essential because it enables practitioners to interpret training behaviour, diagnose failures, select meaningful experiments, and avoid treating a framework’s default settings as scientific justification.

Chapter Summary

A perceptron computes a weighted sum, adds a bias, and applies an activation function. Multilayer networks combine such transformations to learn nonlinear representations. Forward propagation produces predictions, while backpropagation applies the chain rule to calculate gradients. Gradient descent and related optimizers update parameters to reduce a task-specific loss.

Activation functions such as ReLU, sigmoid, tanh, and softmax have different mathematical properties and uses. Good initialization helps preserve useful activation and gradient scales. Regularization methods—including L1/L2 penalties, dropout, and early stopping—reduce overfitting. Input and intermediate normalization can stabilize learning. Hyperparameters must be selected through disciplined validation, and test data must remain isolated until final evaluation. In professional NLP systems, technical performance must be considered alongside data quality, fairness, privacy, transparency, and deployment monitoring.

Key Terms

Perceptron: A basic neuron that applies a weighted sum, bias, and activation.

Weight: A learned coefficient controlling the influence of an input.

Bias: A learned offset added to a weighted sum.

Activation function: A function that transforms a neuron’s pre-activation value.

ReLU: The function max(0, z), widely used in hidden layers.

Sigmoid: An activation producing values between 0 and 1.

Softmax: A function converting logits into a multiclass probability distribution.

Logit: An unnormalised score, often used before sigmoid or softmax.

Forward propagation: Computing network activations from input to output.

Loss function: A measure of disagreement between predictions and targets.

Backpropagation: Efficient computation of parameter gradients using the chain rule.

Gradient descent: An iterative method that updates parameters opposite to the loss gradient.

Learning rate: The step-size hyperparameter used in parameter updates.

Epoch: One complete pass through the training data.

Mini-batch: A subset of training examples used for one update.

Regularization: A method that discourages excessive model complexity.

Dropout: Randomly removing activations during training to improve generalization.

Normalization: Rescaling data or activations to improve numerical and optimization behaviour.

Overfitting: Strong training performance combined with weak generalization.

Data leakage: Unintended use of information from validation, test, or future data during training.

Review Questions

5 Short Answer Questions

1. What role does the bias play in a perceptron?
2. Why are nonlinear activation functions needed in multilayer networks?
3. State one important difference between parameters and hyperparameters.
4. What problem does dropout attempt to reduce?
5. Why should test data be kept separate during hyperparameter selection?

5 Long Answer Questions

1. Explain the structure of a multilayer perceptron and describe the calculation performed at each layer.
2. Compare sigmoid, tanh, ReLU, and softmax in terms of range, use, and limitations.
3. Explain forward propagation, loss calculation, backpropagation, and gradient descent as a complete learning process.
4. Discuss initialization, normalization, and regularization as methods for improving neural-network training.
5. Explain how training and validation curves can be used to diagnose underfitting and overfitting.

3 Analytical / Application Questions

1. Design an evaluation and validation procedure for an NLP sentiment classifier with imbalanced classes. Justify your data split, metrics, and overfitting controls.
2. A model’s training loss decreases steadily, but its validation loss begins rising after the tenth epoch. Analyse at least three possible responses and explain their trade-offs.
3. A neural network receives two numerical features with very different scales. Explain why this may hinder optimization and propose a technically appropriate preprocessing and training strategy.

Multiple Choice Questions with Answers

Question 1

Which component allows a perceptron’s decision boundary to shift away from the origin?

A. The bias

B. The activation derivative

C. The batch size

D. The validation set

Correct Answer: A. The bias

Question 2

Why is a nonlinear activation function needed between linear layers?

A. To reduce the number of training examples

B. To make a stack of layers capable of representing nonlinear functions

C. To guarantee zero training loss

D. To remove all model parameters

Correct Answer: B. To make a stack of layers capable of representing nonlinear functions

Question 3

What is the output range of the sigmoid function?

A. −∞ to ∞

B. −1 to 1

C. 0 to 1

D. 0 to ∞

Correct Answer: C. 0 to 1

Question 4

For binary classification with a sigmoid output, which loss is commonly appropriate?

A. Binary cross-entropy

B. Categorical accuracy only

C. Cosine distance only

D. Mean absolute rank

Correct Answer: A. Binary cross-entropy

Question 5

What does backpropagation compute?

A. The number of classes in the dataset

B. Gradients of the loss with respect to model parameters

C. The final test accuracy without predictions

D. The vocabulary alphabetically

Correct Answer: B. Gradients of the loss with respect to model parameters

Question 6

Using ReLU, what is ReLU(−3)?

A. −3

B. 3

C. 1

D. 0

Correct Answer: D. 0

Question 7

Which method randomly removes activations during training as a form of regularization?

A. Dropout

B. Softmax

C. One-hot encoding

D. Tokenization

Correct Answer: A. Dropout

Question 8

If training loss decreases while validation loss increases, what is the most likely diagnosis?

A. Underflow in the input labels

B. Overfitting

C. Perfect generalization

D. Symmetry breaking

Correct Answer: B. Overfitting

Question 9

Why should normalization statistics normally be computed from the training split only?

A. To prevent information leakage

B. To increase the number of test examples

C. To guarantee a larger vocabulary

D. To eliminate the need for a loss function

Correct Answer: A. To prevent information leakage

Question 10

For a multiclass single-label classifier, what does softmax produce?

A. A set of probabilities that sum to one

B. Only negative values

C. A binary label without scores

D. A set of unnormalised weights that need not be comparable

Correct Answer: A. A set of probabilities that sum to one