Artificial Neurons and Neural Network Architectures

Introduction

Artificial neural networks are computational models inspired by the broad organization of biological nervous systems. They do not reproduce biological brains in detail; rather, they provide mathematical mechanisms for transforming inputs into useful outputs. Modern deep learning systems are built from layers of artificial neurons, adjustable parameters, nonlinear activation functions, and optimization procedures.

This chapter establishes the vocabulary and mathematical foundations needed to study deep learning. It begins with the artificial neuron and perceptron, then develops multilayer feedforward networks, explains parameters and activation functions, introduces the universal approximation idea, and describes computational graphs. The emphasis is on understanding what a network computes, how its architecture affects its capabilities, and why nonlinear activation functions are essential.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the structure and operation of an artificial neuron and perceptron.
2. Distinguish weights, biases, parameters, layers, units, and hyperparameters.
3. Calculate the output of neurons using common activation functions.
4. Explain why multilayer networks can represent functions that a single perceptron cannot.
5. Describe the universal approximation theorem and its limitations.
6. Interpret a feedforward network as a computational graph.
7. Select an appropriate output activation for common prediction tasks.
8. Identify practical, ethical, and engineering limitations of neural network architectures.

1. From Biological Inspiration to Mathematical Models

A biological neuron receives signals through dendrites, integrates them in the cell body, and may transmit a signal through its axon. An artificial neuron abstracts this process into three operations:

• weighted combination of input values;
• addition of a bias;
• nonlinear transformation by an activation function.

For an input vector x = (x₁, x₂, …, xₙ), weight vector w = (w₁, w₂, …, wₙ), and bias b, the pre-activation or net input is

Here, xᵢ and wᵢ are corresponding input and weight values, b is a scalar bias, and z is a scalar. The activation output is

where φ is the activation function. The variables are assumed to be numerical and mutually compatible; if xᵢ represents a measured quantity, preprocessing is often required so that its scale does not dominate learning.

The bias allows the neuron to shift its decision boundary or activation response. Without a bias, a linear decision boundary must pass through the origin, which unnecessarily restricts the model.

2. The Perceptron

The perceptron, introduced by Frank Rosenblatt, is a binary classifier. Its activation is commonly the threshold function:

The output is therefore determined by whether the weighted sum reaches a threshold. The corresponding decision boundary is

For two inputs, this equation describes a straight line; in higher dimensions it describes a hyperplane. Points on one side are classified as one class and points on the other side as the alternative class.

Worked Example: A Two-Input Perceptron

Suppose a perceptron has w₁ = 2, w₂ = −1, b = −0.5, and input x = (1, 0). Then

Since z ≥ 0, the output is 1. For x = (0, 1),

The output is 0. This calculation illustrates classification, not probability. A threshold perceptron does not naturally express confidence.

Linear Separability

A dataset is linearly separable if a single hyperplane can separate its classes without error. AND and OR logic gates are linearly separable. XOR is not linearly separable: no single straight line can place the two positive XOR cases on one side and the two negative cases on the other. Consequently, a single perceptron cannot represent XOR.

The classical perceptron learning rule updates parameters when an example is misclassified. With target y, prediction ŷ, learning rate η > 0, and input x, a commonly used form is

The rule assumes binary targets and a threshold output. It is not the general training rule for modern neural networks. The perceptron convergence theorem states that, under suitable conditions—including linearly separable training data and repeated presentation of examples—the algorithm finds a separating hyperplane. It does not guarantee convergence for nonseparable data, nor does it guarantee the best probabilistic model.

3. Multilayer Feedforward Networks

A multilayer feedforward neural network, often called a multilayer perceptron (MLP), contains an input layer, one or more hidden layers, and an output layer. Information flows from input to output without directed cycles or feedback connections.

A layer computes a vector transformation. For layer l,

where a^(0) = x, W^(l) is the weight matrix, b^(l) is the bias vector, z^(l) is the pre-activation vector, and a^(l) is the activation vector. Matrix dimensions must be compatible. If layer l−1 has n₍ₗ₋₁₎ units and layer l has nₗ units, then W^(l) has dimension nₗ × n₍ₗ₋₁₎ and b^(l) has dimension nₗ.

A network with L layers can be written as the composition

The term deep generally refers to a model with multiple learned transformation layers, although there is no universally fixed numerical threshold for the word “deep.”

Parameters and Hyperparameters

Parameters are values learned from data, principally weights and biases. Hyperparameters are design or training choices specified externally, such as learning rate, batch size, number of layers, number of hidden units, activation type, and regularization strength.

For a fully connected layer with n inputs and m neurons, the number of trainable parameters is

The first term counts weights and the second counts biases. For example, a layer with 4 inputs and 3 neurons has 4 × 3 + 3 = 15 trainable parameters.

Worked Example: A Small Network

Consider two inputs, one hidden layer containing two neurons, and one linear output neuron. Let

W^(1) = [[1, −1], [0.5, 2]], b^(1) = [0, 1], x = [2, 1].

The hidden pre-activation is

Using ReLU, φ(z) = max(0, z), the hidden activation is a^(1) = [1, 4]. If the output weights are [2, −0.5] and output bias is 1, then

This example demonstrates forward propagation: values are computed sequentially from the input layer to the output layer.

4. Activation Functions

Without nonlinear activation functions, stacking linear layers does not create a more expressive model. If f(x) = A x + c and g(x) = Bx + d, then

which is still an affine transformation. Thus, nonlinear activations are essential in hidden layers.

Step Function

The step function is useful for describing the perceptron, but it is discontinuous and has zero derivative almost everywhere. This makes it unsuitable for gradient-based training of multilayer networks.

Sigmoid

The sigmoid maps any real input to (0, 1). It is useful for a binary output interpreted as a probability, especially when paired with binary cross-entropy. Its derivative is

For very large positive or negative z, the function saturates and its derivative becomes small, contributing to vanishing gradients.

Hyperbolic Tangent

The tanh function maps values to (−1, 1) and is zero-centred. It can still saturate for large absolute inputs.

ReLU

ReLU is computationally simple and often effective in hidden layers. Its derivative is 1 for z > 0 and 0 for z < 0; at z = 0, a subgradient or implementation convention is used. A unit that remains in the negative region may stop receiving useful gradient updates, a phenomenon called the dying-ReLU problem.

Leaky ReLU and Softmax

A leaky ReLU preserves a small negative slope:

where α is a small positive constant, such as 0.01, selected as a design choice. For a K-class output, softmax converts logits z₁, …, zₖ into values summing to 1:

The outputs can be interpreted as a categorical probability distribution when the model and loss are appropriately specified. Numerically stable implementations subtract max(z) before exponentiation; this does not change the resulting probabilities.

5. Output Layers and Prediction Tasks

Architecture should reflect the target variable.

• For regression, a linear output is common: ŷ = z. The output is unbounded and is often trained with mean squared error or a related loss.
• For binary classification, one sigmoid output is common. Its value can represent an estimated probability under appropriate modelling assumptions.
• For mutually exclusive multiclass classification, K logits followed by softmax are common.
• For multilabel classification, where several labels may be true simultaneously, use one sigmoid per label rather than a single softmax.

A neural network’s output is not automatically a calibrated probability. Calibration, class imbalance, sampling design, and data quality must be evaluated separately.

6. Universal Approximation

The universal approximation theorem states, in one common formulation, that a feedforward network with a suitable nonlinear activation and at least one hidden layer can approximate any continuous function on a compact subset of finite-dimensional Euclidean space to arbitrary accuracy, given enough hidden units.

The statement is powerful but frequently misunderstood. It does not say that:

• a small network can approximate every function;
• training will find the required parameters;
• the approximation will generalize outside the training distribution;
• the theorem applies without conditions to discontinuous or unbounded functions;
• one hidden layer is computationally preferable to a deep architecture.

“Arbitrary accuracy” refers to the existence of parameters under mathematical conditions, not necessarily practical learnability. Depth can express some hierarchical functions more efficiently than a very wide shallow network. In practice, architecture, optimization, data quantity, inductive bias, and regularization all influence performance.

7. Computational Graphs and Forward Propagation

A computational graph represents a calculation as nodes and directed edges. Nodes perform operations, while edges carry intermediate values. A simple neuron can be represented as multiplication, addition, and activation operations:

Figure: A computational graph for one artificial neuron.

The graph clarifies dependencies and supports automatic differentiation. During backpropagation, derivatives are calculated using the chain rule. If a loss J depends on a through a = φ(z), then

For z = wx + b,

These equations describe local derivatives; actual values depend on the loss, inputs, parameters, and activation. Automatic differentiation applies this principle systematically through large graphs.

8. A Detailed Applied Case: Loan-Default Risk Screening

Illustrative case: A financial institution wants to estimate whether an applicant may default within twelve months. The purpose is to demonstrate architecture and governance, not to prescribe a real lending system.

Background and Problem

The institution has historical records containing income, debt-to-income ratio, repayment history, employment duration, and other attributes. The target is binary: default or no default. The organization requires a model that is accurate, explainable enough for review, and monitored for discriminatory outcomes.

Analysis and Approach

A baseline logistic model should first be established. An MLP might then use normalized numerical features, a small hidden layer with ReLU units, and one sigmoid output. The data must be divided into training, validation, and test sets without leakage. If future information accidentally enters the features, test performance will be misleading.

The output probability should not be treated as a decision automatically. A threshold should be selected using the costs of false approvals and false rejections, while also considering legally protected groups. Class imbalance may require weighted loss, resampling, or threshold analysis.

Outcome and Governance

Suppose the neural model improves ranking performance on a temporally separated test set but produces unequal error rates across groups. The appropriate response is not simply to deploy the model because its average accuracy is higher. The organization should investigate data quality, feature validity, threshold effects, and alternative models. It should document model purpose, training data, limitations, monitoring metrics, human review procedures, and appeal mechanisms.

Learning Points

1. Architecture must match the prediction task.
2. A neural output is only one component of a decision system.
3. Validation must reflect future deployment conditions.
4. Fairness, privacy, explainability, and accountability are engineering requirements, not optional additions.
5. A simpler model may be preferable when performance is comparable and interpretability is important.

9. Architectural Choices and Professional Practice

Fully connected networks are suitable for tabular vectors and general function approximation, but they do not automatically exploit spatial, temporal, or relational structure. Convolutional networks incorporate local connectivity and shared parameters for grid-like data. Recurrent and attention-based architectures are designed for sequential or contextual relationships. These architectures are studied in later chapters, but the same foundational concepts—parameters, transformations, nonlinearities, and computational graphs—remain applicable.

Good practice includes:

• standardizing or otherwise preprocessing features appropriately;
• choosing activations according to the layer’s role;
• tracking parameter counts and tensor dimensions;
• separating training, validation, and test data;
• checking for leakage and distribution shift;
• using reproducible experiments and documented random seeds where appropriate;
• comparing with simple baselines;
• monitoring calibration, subgroup performance, and operational failure modes;
• avoiding unnecessary architectural complexity.

Limitations

Neural networks can overfit, especially when the model has many parameters relative to the information in the data. They may be sensitive to feature scaling, initialization, optimization settings, noisy labels, and distribution changes. High predictive accuracy does not establish causality. Correlation learned from historical data can reproduce institutional bias or fail under changed conditions. Adversarial inputs, privacy leakage, and excessive confidence are additional concerns.

Contemporary Perspective

Current neural network engineering increasingly emphasizes efficient architectures, transfer learning, pretrained representations, quantization, pruning, uncertainty estimation, and responsible deployment. Large models may contain billions of parameters, but parameter scale alone does not guarantee reliability. Evaluation should include robustness, safety, data governance, energy and infrastructure costs, and performance on the populations affected by the system.

Chapter Summary

An artificial neuron computes a weighted sum, adds a bias, and applies an activation function. A perceptron uses a threshold and can represent only linearly separable decision rules. Multilayer feedforward networks compose affine transformations and nonlinear activations, allowing them to model complex functions such as XOR. Weights and biases are learned parameters, while architectural and training choices are hyperparameters.

Activation functions determine the transformation and gradient behaviour of neurons. Sigmoid and softmax are commonly associated with probabilistic outputs, while ReLU and related functions are widely used in hidden layers. The universal approximation theorem establishes an existence result for approximating continuous functions, but it does not guarantee efficient training, generalization, or reliable deployment.

Computational graphs make forward computation explicit and provide the structure for backpropagation through the chain rule. In professional applications, architecture must be combined with sound data practices, baseline comparisons, monitoring, fairness assessment, privacy protection, and human governance.

Key Terms

Artificial neuron: A computational unit that applies a weighted sum, bias, and activation function.

Activation function: A function applied to a neuron’s pre-activation.

Bias: A trainable offset that shifts a neuron’s response.

Computational graph: A directed representation of variables and operations in a calculation.

Deep learning: Machine learning based largely on multilayer neural networks.

Feedforward network: A network in which information flows without directed cycles.

Hyperparameter: A design or training setting selected outside parameter learning.

Linear separability: The possibility of separating classes with a hyperplane.

Logit: An unnormalized output value, often supplied to softmax or sigmoid.

MLP: Multilayer perceptron; a feedforward network with fully connected layers.

Parameter: A learned weight or bias.

Perceptron: A threshold-based linear binary classifier.

Pre-activation: The weighted sum plus bias before activation.

ReLU: The function max(0, z).

Sigmoid: A function mapping real values to the interval (0, 1).

Softmax: A function converting a vector of logits into positive values summing to one.

Universal approximation theorem: A result concerning approximation of continuous functions by suitable neural networks.

Weight: A learned coefficient controlling an input’s contribution.

Review Questions

Short Answer Questions

1. What roles do weights and biases play in an artificial neuron?
2. Why can a single perceptron not represent XOR?
3. Distinguish a parameter from a hyperparameter.
4. Why are nonlinear activation functions necessary in multilayer networks?
5. What is the purpose of a computational graph?

Long Answer Questions

1. Explain the structure and operation of a multilayer feedforward network, including the dimensions of its weight matrices.
2. Compare sigmoid, tanh, ReLU, and softmax in terms of range, use, and limitations.
3. State the universal approximation theorem carefully and explain at least three things it does not guarantee.
4. Describe forward propagation and explain how backpropagation uses the chain rule.
5. Discuss the principal risks and governance requirements when deploying a neural network for high-impact decisions.

Analytical / Application Questions

1. Design a suitable output layer and activation function for each of these tasks: house-price regression, mutually exclusive image classification, and multilabel medical coding. Justify each choice.
2. A fully connected layer has 8 inputs and 5 neurons. Calculate its number of trainable parameters and explain each component of the calculation.
3. A model performs well on an overall test set but has substantially higher false-negative rates for one demographic group. Analyze possible causes and propose a responsible evaluation and remediation process.

Multiple Choice Questions with Answers

Question 1

For an artificial neuron, what is the pre-activation when w = [2, −1], x = [3, 2], and b = 0.5?

A. 4.5

B. 6.5

C. −4.5

D. 1.5

Correct Answer: A. 4.5

Question 2

Which problem cannot be solved by a single standard perceptron?

A. AND classification

B. OR classification

C. XOR classification

D. A linearly separable classification problem

Correct Answer: C. XOR classification

Question 3

How many trainable parameters are in a fully connected layer with 8 inputs and 5 neurons?

A. 13

B. 40

C. 45

D. 64

Correct Answer: C. 45

Question 4

Which activation function is most commonly used to produce probabilities for mutually exclusive K-class classification?

A. ReLU

B. Softmax

C. tanh

D. Step function

Correct Answer: B. Softmax

Question 5

Why is a nonlinear activation required between affine layers?

A. It reduces the number of inputs to zero

B. It makes every output a probability

C. It prevents a composition of layers from collapsing into one affine transformation

D. It guarantees perfect generalization

Correct Answer: C. It prevents a composition of layers from collapsing into one affine transformation

Question 6

Which statement about the universal approximation theorem is correct?

A. It guarantees that training always finds the best parameters

B. It states that suitable networks can approximate certain continuous functions under specified conditions

C. It proves that shallow networks are always better than deep networks

D. It applies only to binary classifiers

Correct Answer: B. It states that suitable networks can approximate certain continuous functions under specified conditions

Question 7

What is the principal computational purpose of backpropagation?

A. To choose the test set

B. To calculate gradients through the computational graph

C. To convert all outputs into integers

D. To remove all model parameters

Correct Answer: B. To calculate gradients through the computational graph

Question 8

Which output activation is commonly appropriate for binary classification?

A. One sigmoid unit

B. A linear unit with no loss

C. A ReLU for each mutually exclusive class

D. A step function trained by ordinary gradient descent

Correct Answer: A. One sigmoid unit

Question 9

For ReLU(z) = max(0, z), what is the output when z = −3?

A. −3

B. 3

C. 0

D. 1

Correct Answer: C. 0

Question 10

Which statement best describes a bias in a neuron?

A. It is always fixed at zero

B. It shifts the neuron’s response and is usually learned

C. It determines the size of the training dataset

D. It replaces all weights

Correct Answer: B. It shifts the neuron’s response and is usually learned