Neural Networks and Contemporary AI

Introduction and Context

Neural networks are computational models inspired loosely by networks of biological neurons. In machine learning, they are used to learn a relationship between input variables and an output by adjusting numerical parameters from data. Their importance has grown because modern computing hardware, large datasets, and improved optimization methods make it possible to train networks with millions or billions of parameters.

Neural networks support applications such as image classification, speech recognition, recommendation, fraud detection, language translation, medical-image analysis, forecasting, and generative artificial intelligence. However, a neural network is not automatically intelligent or reliable. Its performance depends on data quality, model design, training procedures, evaluation, deployment conditions, and governance.

This chapter develops the progression from the perceptron to multilayer networks, explains activation functions and backpropagation, outlines a practical deep-learning workflow, and introduces transformers, foundation models, and generative AI. It also examines limitations including bias, overfitting, hallucination, lack of interpretability, security risks, and computational cost.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the structure and operation of a perceptron.
2. Distinguish linear models from multilayer neural networks.
3. Describe common activation functions and their practical uses.
4. Explain forward propagation, loss functions, gradient descent, and backpropagation.
5. Design a basic deep-learning workflow for a data-science problem.
6. Describe the roles of convolutional networks, recurrent networks, and transformers.
7. Explain foundation models, transfer learning, prompting, fine-tuning, and retrieval-augmented generation.
8. Evaluate contemporary AI systems critically, including their technical, ethical, and operational limitations.

1. The Perceptron: A Basic Neural Unit

A perceptron accepts numerical inputs, multiplies each by a weight, adds a bias, and applies a decision rule. For inputs x₁, x₂, …, xₚ, weights w₁, w₂, …, wₚ, and bias b, the weighted sum is:

Here, z is a real-valued pre-activation score. The perceptron produces a binary output using a threshold function:

The inputs and weights must be numerically compatible. For example, if xᵢ represents standardized measurements, the weights correspond to the standardized scale. The bias shifts the decision boundary and is not multiplied by an input feature.

The decision boundary is given by:

For two input variables, this is a straight line; for three variables, a plane; and in higher dimensions, a hyperplane. The perceptron can solve linearly separable classification problems, such as separating two clearly separated groups. It cannot solve the classic exclusive-OR (XOR) problem using a single unit because XOR classes cannot be separated by one straight boundary.

The perceptron learning rule updates parameters when a prediction is incorrect. A general form is:

where y is the target, ŷ is the predicted binary output, and η is a positive learning rate. This rule is appropriate for the threshold-based perceptron and is not the same as the gradient update used by every modern neural network.

A perceptron is historically important because it establishes the central idea of learning parameters from examples. Its major limitation is that a single layer can represent only a linear decision function.

2. From Single Layers to Multilayer Networks

A multilayer perceptron (MLP) contains an input layer, one or more hidden layers, and an output layer. Neurons in adjacent layers are commonly fully connected. Each neuron performs a weighted sum followed by an activation function.

For hidden layer l, the vector form is:

Here, a⁽ˡ⁻¹⁾ is the output from the preceding layer, W⁽ˡ⁾ is the weight matrix, b⁽ˡ⁾ is the bias vector, and φ⁽ˡ⁾ is applied element by element. The input is often denoted a⁽⁰⁾ = x. The final output may be a probability, a real-valued prediction, or a vector of class scores.

The expressive power of an MLP comes from composition. One layer transforms the representation, and later layers transform it again. With suitable nonlinear activations and sufficient capacity, multilayer networks can approximate a broad class of continuous functions. This does not mean that every network learns a useful function: optimization, data coverage, architecture, and regularization remain essential.

A network with no nonlinear activation between layers collapses mathematically into a linear transformation. For example, two consecutive linear transformations satisfy:

Therefore, nonlinear activation functions are necessary if a deep network is to represent nonlinear relationships.

Parameters and capacity

The number of trainable parameters in a fully connected layer with n inputs and m neurons is:

The first term counts weights and the second counts biases. A large parameter count can allow complex representations but can also increase memory use, training time, overfitting risk, and security exposure. Capacity should be matched to the amount and quality of available data.

3. Activation Functions

An activation function determines how a neuron transforms its pre-activation value. It introduces nonlinearity and influences gradient flow during training.

Step function

The step function is suitable for the classical perceptron but is unsuitable for ordinary gradient-based training because it is discontinuous and has zero derivative almost everywhere.

Sigmoid

The sigmoid function maps a real value to the interval (0, 1):

Its derivative is:

A sigmoid output is useful for a binary classification output when interpreted as an estimated probability, assuming the model and calibration are appropriate. In hidden layers, sigmoid units can suffer from vanishing gradients when z is very positive or very negative.

Hyperbolic tangent

The hyperbolic tangent maps values to (−1, 1):

It is zero-centred, but it can also saturate and produce small gradients at extreme inputs.

ReLU

The rectified linear unit is:

Its derivative is 1 for z > 0 and 0 for z < 0; at z = 0, implementations choose a convention or use a subgradient. ReLU is computationally simple and often effective in hidden layers. A unit can become permanently inactive if it remains in the negative region, a problem sometimes called the dying-ReLU issue.

Variants and output activations

Leaky ReLU assigns a small negative slope rather than zero for negative inputs. Softmax converts a vector of logits into values that sum to one:

It is commonly used for mutually exclusive multiclass classification. For numerical stability, implementations usually subtract the maximum logit before exponentiation; this does not change the resulting probabilities.

A regression output often uses a linear activation. A binary classifier commonly uses a sigmoid output, while a multiclass classifier commonly uses softmax. These choices must match the target representation and loss function.

4. Forward Propagation, Loss, and Learning

During forward propagation, the network calculates an output from an input by moving through its layers. Training compares the output with the target through a loss function.

For one binary observation with target y ∈ {0, 1} and predicted probability p, binary cross-entropy is:

For n observations, the mean loss is:

The logarithm requires p to lie strictly between 0 and 1. Practical software clips probabilities for numerical safety. Cross-entropy strongly penalizes confident incorrect predictions.

For regression, mean squared error is:

MSE has squared units of the target variable. It is sensitive to large errors and is appropriate when such sensitivity is desired and the target scale is meaningful.

The training objective is to find parameters θ that minimize a loss J(θ). Gradient descent updates parameters as follows:

Here, η is the learning rate and ∇θJ is the gradient of the loss with respect to all trainable parameters. The gradient points in the direction of greatest increase, so subtraction moves toward local decrease. In practice, gradients are often estimated using a mini-batch rather than the entire dataset.

5. Backpropagation

Backpropagation is an efficient application of the chain rule for calculating gradients through a layered computational graph. It does not itself specify the optimizer; it supplies derivatives that optimizers such as stochastic gradient descent or Adam can use.

Consider a simple output layer:

If the loss is L(a, y), the chain rule gives:

Since ∂z ÷ ∂w = x, the input directly scales the parameter gradient. Similarly, ∂z ÷ ∂b = 1, so:

For a multilayer network, the output error is propagated backward. Each layer receives an error signal based on the downstream error, the next layer's weights, and the derivative of its activation. The resulting gradients are accumulated over a batch and used to update weights and biases.

Worked example: one neuron

Suppose a neuron has x = 2, w = 0.4, b = −0.1, uses a linear activation, and has target y = 1.5. The prediction is:

Using one-observation squared error defined as L = ½(ŷ − y)², the error is −0.8 and:

With learning rate η = 0.1, the updates are:

The updated prediction for x = 2 is 1.10. It has moved closer to 1.5, although one update does not guarantee optimality. This example assumes a single data point, a linear output, and the specified half-squared-error convention.

6. A Practical Deep-Learning Workflow

A professional workflow begins with the problem rather than the model.

6.1 Define the task and target

Clarify whether the task is classification, regression, ranking, forecasting, representation learning, or generation. Specify the unit of prediction, prediction time, acceptable error, operational cost of false positives and false negatives, and success metric.

6.2 Prepare and inspect data

Check missing values, duplicates, outliers, inconsistent labels, leakage, class imbalance, and changing data distributions. Split data into training, validation, and test sets. The test set should remain untouched until final evaluation. For time-dependent data, use chronological splits rather than random splits when random splitting would allow future information into training.

Numerical features are often standardized using training-set statistics:

where μ_train is the training mean and σ_train is the training standard deviation. These statistics must not be calculated using validation or test observations, because that would leak information.

6.3 Establish a baseline

A simple rule, linear model, decision tree, or previously deployed system provides a reference. A neural network should be retained only if it provides meaningful improvement, considering cost, latency, interpretability, and maintenance.

6.4 Design the model

Choose architecture, layer widths, activation functions, output encoding, loss, and regularization. Common techniques include weight decay, dropout, early stopping, data augmentation, and batch normalization. Each addresses a different problem and should be validated rather than added automatically.

6.5 Train and monitor

Track training and validation loss, relevant task metrics, learning rate, computational resource use, and failures. If training loss decreases while validation loss rises, the model may be overfitting. If both remain high, the model may be underpowered, poorly optimized, incorrectly labelled, or trained on inadequate features.

6.6 Evaluate beyond one score

For classification, inspect confusion matrices, precision, recall, F1 score, calibration, and subgroup performance. Accuracy is misleading for highly imbalanced data. For regression, use measures such as MAE, RMSE, and error by relevant segment. Always examine examples of success and failure.

6.7 Deploy and monitor

Deployment requires versioned code, data and model artefacts; reproducible preprocessing; access control; logging; rollback procedures; and monitoring for data drift, concept drift, latency, outages, and fairness changes. A model is a socio-technical system, not merely a saved file.

7. Important Neural-Network Architectures

Convolutional neural networks

Convolutional neural networks (CNNs) use local filters whose parameters are shared across spatial locations. This reduces parameter counts and exploits locality, making CNNs effective for images, video, spatial measurements, and some audio representations. Pooling or strided operations can reduce spatial resolution while retaining salient patterns.

Recurrent neural networks

Recurrent neural networks process sequences by maintaining a hidden state. They can model order but may struggle with long-range dependencies because repeated multiplication can cause vanishing or exploding gradients. Long short-term memory and gated recurrent units address some of these problems through gating mechanisms.

Attention and transformers

Attention allows a model to assign different weights to elements of an input when constructing a representation. In scaled dot-product self-attention, query, key, and value matrices are used:

Q, K, and V are query, key, and value matrices, and d_k is the key-vector dimension. The square-root scaling helps control the magnitude of dot products, supporting stable softmax gradients. Self-attention relates tokens within a sequence and can be computed in parallel, unlike strictly sequential recurrence. Positional information is added because attention alone does not inherently encode order.

Transformers typically combine multi-head attention, feed-forward networks, residual connections, normalization, and positional representations. They have become central to language models and are also used in vision, audio, biological sequences, and multimodal systems. Their computational and memory cost can be substantial for very long sequences.

8. Foundation Models and Generative AI

A foundation model is a large model trained on broad data and adapted to many downstream tasks. Pretraining may involve predicting missing or next elements, learning image-text alignment, or other self-supervised objectives. Adaptation can use prompting, supervised fine-tuning, parameter-efficient fine-tuning, or retrieval-augmented generation (RAG).

Generative AI produces text, images, audio, video, code, or other outputs. A language model commonly represents a sequence as tokens and estimates a conditional distribution:

During generation, tokens are selected repeatedly from predicted probabilities. Greedy decoding chooses the highest-probability token; sampling introduces controlled variation. Temperature modifies the distribution, but it does not make a model factually reliable.

In RAG, a system retrieves relevant documents and supplies them as context to a generative model. This can improve access to current or organization-specific information, but retrieval quality, document permissions, context limits, citation accuracy, and prompt injection remain concerns. RAG is not a guarantee that generated claims are correct.

Fine-tuning changes model parameters using task-specific examples. Prompting changes instructions or context without necessarily changing parameters. Parameter-efficient methods update a small number of additional parameters, reducing cost but not eliminating the need for careful evaluation.

9. Illustrative Case: Predictive Maintenance in a Manufacturing Plant

Background

A factory operates electric motors whose unexpected failure stops production. Sensors record vibration, temperature, current, and operating speed at regular intervals.

Problem

The engineering team wants early warnings of likely failure while avoiding excessive false alarms. A false negative can cause costly downtime; a false positive can cause unnecessary inspection or replacement.

Analysis

The data-science team first defines the prediction horizon: whether the motor will fail within the next seven days. They create labels using maintenance records and ensure that features used at prediction time do not include information recorded after failure. A random split is rejected because adjacent readings from the same motor could appear in both training and test data. Instead, the team uses time-based and motor-aware validation.

Approach and solution

A logistic-regression baseline is established. A small MLP is then trained on standardized sensor summaries, while a one-dimensional CNN is evaluated on short vibration sequences. The team uses a validation set to select thresholds according to the relative costs of missed failures and false alarms. They assess recall, precision, calibration, lead time, and performance by motor type and operating condition.

The final system does not automatically order replacement parts. It sends a risk score to maintenance personnel, displays the main contributing sensor trends, and requires a human decision. The model, preprocessing pipeline, and threshold are versioned. Performance is monitored because new motor models and seasonal operating conditions may cause drift.

Outcome

In this illustrative scenario, the neural model is retained only if its improvement over the baseline justifies additional complexity. The outcome is not judged by accuracy alone: usable warning time, maintenance workload, subgroup reliability, and cost savings are also considered.

Learning points

The case demonstrates that architecture selection follows problem definition, data design, and operational constraints. It also illustrates why leakage prevention, temporal validation, cost-sensitive evaluation, human oversight, and monitoring are as important as backpropagation.

10. Limitations, Risks, and Governance

Data and statistical limitations

Neural networks learn patterns present in their training data. They may fail under distribution shift, rare events, noisy labels, missing subgroups, or changing relationships. High test performance can result from leakage or an unrepresentative test set. Correlation learned by a network should not automatically be interpreted as causation.

Interpretability and uncertainty

Many neural networks are difficult to explain in terms understandable to users. Feature-attribution methods can be useful but are not perfect causal explanations. A confidence score is not necessarily a calibrated probability. High-stakes systems should use independent validation, uncertainty analysis where appropriate, human review, and clear escalation procedures.

Bias and fairness

Bias can enter through historical labels, sampling, measurement processes, proxy variables, or deployment decisions. Evaluation should examine relevant subgroups and error trade-offs. There is no universal fairness criterion that solves every social context; organizations must identify affected groups, legal obligations, harms, and acceptable trade-offs.

Privacy and security

Training data may contain personal or confidential information. Risks include memorization, unauthorized inference, model extraction, adversarial examples, data poisoning, and prompt injection. Controls include data minimization, access restrictions, secure pipelines, red-team testing, monitoring, and incident response.

Generative-AI-specific risks

Generative systems may hallucinate plausible but false statements, reproduce copyrighted or private content, generate unsafe material, expose confidential prompts, or amplify stereotypes. They can also produce code with vulnerabilities. Human review, provenance checks, retrieval validation, output filtering, and policies for sensitive use are necessary.

Environmental and economic costs

Training and serving large models consume energy, hardware, and financial resources. Model selection should consider whether a smaller model, retrieval system, traditional algorithm, or human process is sufficient. Efficiency techniques include distillation, quantization, caching, sparse computation, and careful request management.

Professional best practices

A responsible neural-network project should:

• define a measurable objective and decision context;
• use leakage-resistant data splits and a strong baseline;
• document data sources, assumptions, model versions, and limitations;
• evaluate subgroup performance and operational costs;
• protect personal, confidential, and copyrighted information;
• test robustness, security, and failure modes;
• keep humans accountable for consequential decisions;
• monitor performance after deployment and provide rollback mechanisms.

11. Contemporary Direction of the Field

Current AI development emphasizes multimodal models, efficient adaptation, on-device inference, agentic workflows, synthetic data, retrieval systems, and specialized models. These developments expand capability but also increase system complexity. An AI agent that calls tools, accesses databases, or takes actions creates risks beyond those of a passive predictor, including authorization failures and compounding errors.

The central professional skill is therefore not simply constructing a larger model. It is selecting an appropriate method, validating it against the real task, communicating uncertainty, and governing its use. Data scientists must combine mathematical understanding with software engineering, domain knowledge, ethics, and risk management.

Chapter Summary

Neural networks learn parameterized transformations from data. A perceptron forms a weighted sum and applies a decision rule, but a single layer is limited to linear separation. Multilayer networks gain expressive power through composition and nonlinear activation functions. Sigmoid, tanh, ReLU, and softmax have different mathematical properties and practical roles.

Forward propagation generates predictions; a loss function measures error; backpropagation applies the chain rule to calculate gradients; and gradient-based optimization updates parameters. A reliable deep-learning workflow includes precise problem definition, leakage-resistant data preparation, baselines, suitable architecture, validation, multiple evaluation metrics, deployment controls, and monitoring.

CNNs exploit local structure, recurrent models represent sequences through hidden states, and transformers use attention to relate elements across a context. Foundation models are broadly pretrained systems that can be adapted through prompting, fine-tuning, or retrieval. Generative AI produces new content but can hallucinate, reproduce bias, leak information, and create security or copyright risks.

Effective contemporary AI practice is therefore both technical and organizational. Model capability must be balanced against accuracy, fairness, privacy, interpretability, cost, environmental impact, safety, and accountability.

Key Terms

Perceptron: A single computational unit that applies a weighted sum and decision function.

Weight: A trainable parameter that scales an input or intermediate signal.

Bias: A trainable offset added to a weighted sum.

Activation function: A function that transforms a neuron's pre-activation value.

Multilayer perceptron: A feed-forward network containing one or more hidden layers.

Linear separability: The property that classes can be separated by a hyperplane.

Forward propagation: Computing a network output by moving information from input to output.

Loss function: A function that quantifies disagreement between predictions and targets.

Gradient descent: An optimization method that updates parameters opposite the loss gradient.

Backpropagation: Efficient calculation of parameter gradients using the chain rule.

Overfitting: Learning patterns that improve training performance but fail to generalize.

Regularization: A method that discourages undesirable complexity or improves generalization.

Convolutional neural network: A network using local, shared filters, commonly for spatial data.

Recurrent neural network: A sequence model that maintains a hidden state over time.

Attention: A mechanism that weights relationships among input elements.

Transformer: An architecture built substantially around attention and feed-forward components.

Foundation model: A broadly pretrained model adaptable to many tasks.

Generative AI: AI that produces new content such as text, images, audio, or code.

Hallucination: A plausible-looking but unsupported or false generated output.

Retrieval-augmented generation: Generation supported by retrieved external information.

Review Questions

Short Answer Questions

1. What limitation does a single perceptron have in relation to nonlinear classification?
2. Why are nonlinear activation functions necessary in a multilayer network?
3. What is the purpose of a loss function during neural-network training?
4. How does backpropagation support gradient-based learning?
5. State two risks associated with generative AI.

Long Answer Questions

1. Explain the structure of a multilayer perceptron and describe the roles of weights, biases, activations, and output layers.
2. Compare sigmoid, tanh, ReLU, and softmax, including suitable uses and important limitations.
3. Describe a complete deep-learning workflow from problem definition through post-deployment monitoring.
4. Explain scaled dot-product attention and discuss why transformers have become important in contemporary AI.
5. Discuss the technical, ethical, privacy, and governance limitations of foundation models.

Analytical / Application Questions

1. A bank wants to predict loan default using customer data. Propose an evaluation design that addresses class imbalance, fairness, leakage, calibration, and the costs of false decisions.
2. A factory has only 2,000 labelled images of defective products but access to a large pretrained vision model. Recommend an adaptation strategy and identify the tests needed before deployment.
3. An organization proposes a chatbot that answers questions from internal policy documents. Analyze whether retrieval-augmented generation is appropriate and specify controls for outdated documents, access permissions, hallucinations, prompt injection, and human escalation.

Multiple Choice Questions with Answers

Question 1

What is the main limitation of a single perceptron?

A. It cannot store any numerical weights

B. It can represent only a linear decision boundary

C. It always requires image data

D. It cannot use a bias term

Correct Answer: B. It can represent only a linear decision boundary

Question 2

Which activation function is defined as max(0, z)?

A. Sigmoid

B. Tanh

C. ReLU

D. Softmax

Correct Answer: C. ReLU

Question 3

In gradient descent, parameters are generally updated by:

A. Adding the loss gradient multiplied by the learning rate

B. Subtracting the loss gradient multiplied by the learning rate

C. Replacing all parameters with zero

D. Sorting parameters by magnitude

Correct Answer: B. Subtracting the loss gradient multiplied by the learning rate

Question 4

What does backpropagation calculate efficiently?

A. Database indexes

B. Parameter gradients using the chain rule

C. The number of training records

D. The final deployment cost

Correct Answer: B. Parameter gradients using the chain rule

Question 5

Which output activation is commonly used for mutually exclusive multiclass classification?

A. Softmax

B. Linear only

C. Step function only

D. ReLU only

Correct Answer: A. Softmax

Question 6

Why should validation and test data not be used to calculate training standardization statistics?

A. It causes information leakage

B. It prevents matrix multiplication

C. It makes all features categorical

D. It removes the model bias term

Correct Answer: A. It causes information leakage

Question 7

What is a key feature of convolutional neural networks?

A. They use local filters with shared parameters

B. They cannot process numerical inputs

C. They always require recurrent hidden states

D. They use no trainable parameters

Correct Answer: A. They use local filters with shared parameters

Question 8

What is the primary operation emphasized by transformer architectures?

A. Manual feature selection

B. Self-attention

C. Random guessing

D. Database normalization

Correct Answer: B. Self-attention

Question 9

What is a common limitation of generative language models?

A. They cannot produce token sequences

B. They may generate plausible but unsupported claims

C. They never require training data

D. They always provide calibrated probabilities

Correct Answer: B. They may generate plausible but unsupported claims

Question 10

Which practice best supports responsible deployment of a high-stakes neural model?

A. Evaluate only training accuracy

B. Remove all human oversight

C. Monitor drift, subgroup performance, and failure modes

D. Use the largest possible model regardless of cost

Correct Answer: C. Monitor drift, subgroup performance, and failure modes