1. Foundations of Generative AI

Introduction and Context

Generative artificial intelligence refers to computational systems that learn patterns from data and use those patterns to produce new content. The content may be text, images, audio, video, software code, molecular structures, or synthetic tabular records. Unlike a system designed only to assign an input to a predefined class, a generative system attempts to model how data could have been produced. This distinction is central to understanding large language models, image generators, multimodal systems, and many modern AI applications.

Generative AI is not simply a collection of tools for producing realistic outputs. It is a family of statistical modelling approaches concerned with probability distributions, representation learning, uncertainty, and sampling. A model trained on documents may learn regularities in syntax, semantics, discourse, and factual associations. A model trained on images may learn relationships among shapes, textures, colours, and objects. Generation occurs when the learned structure is used to construct a new sample rather than merely recognize an existing one.

The foundations developed in this chapter support later study of language-model architecture, pre-training, fine-tuning, prompting, alignment, retrieval augmentation, evaluation, and deployment. A postgraduate student should therefore be able to distinguish the principal model families, interpret their objectives, understand their assumptions, and assess their practical trade-offs.

Learning Objectives

After studying this chapter, the reader should be able to:

1. Distinguish generative learning from discriminative learning using probabilistic reasoning.

2. Explain probability distributions, likelihood, sampling, uncertainty, and latent representations.

3. Describe the operating principles of autoregressive models, variational autoencoders, generative adversarial networks, diffusion models, and transformers.

4. Relate model objectives to practical applications and engineering choices.

5. Analyse the limitations, risks, and governance concerns associated with generative systems.

6. Select an appropriate model family for a given business or technical scenario.

From Recognising Data to Modelling Its Possibilities

Suppose a dataset contains examples represented by a variable x and, where relevant, labels represented by y. A discriminative model focuses on the conditional distribution p(y|x), or on a decision boundary that separates classes. For example, a medical classifier may estimate whether an image indicates a particular disease. It need not generate a medically plausible image of a new patient.

A generative model seeks to learn a distribution involving the data itself, commonly p(x), or the joint distribution p(x,y). Once such a distribution has been learned, the system can estimate which observations are plausible and can sample new observations. A generative classifier can also use p(x|y) together with p(y) to infer p(y|x), but the defining emphasis is modelling how observations arise rather than only separating categories.

The distinction is useful but not absolute. Many modern systems combine generative and discriminative components. A language model generates tokens, while a separate classifier may evaluate safety. A diffusion model generates an image, while a vision encoder may provide conditioning information. The important question is the learning objective and the role played by probability, not whether an application is labelled “generative.”

Probability Distributions and Data Generation

A probability distribution assigns probabilities to possible outcomes. In a discrete setting, the probabilities of all outcomes sum to one. In a continuous setting, a probability density describes relative likelihood, and probabilities are obtained over intervals. A model attempts to represent a distribution sufficiently well that samples from it resemble samples from the training population.

For a sequence of tokens x = (x1, x2, ..., xT), the chain rule of probability gives:

p(x1, x2, ..., xT) = p(x1)p(x2|x1) ... p(xT|x1, ..., xT−1).

This identity is fundamental to autoregressive language modelling. The model does not need to predict a complete document in one operation. It predicts the next token conditional on the preceding context, and repeated sampling produces a sequence.

A learned distribution is always an approximation. Training data may be incomplete, biased, duplicated, noisy, or drawn from a population different from the deployment population. Consequently, probability estimates reflect both the model and the data-generating process represented in the dataset. A fluent output is not automatically a true, fair, or useful output.

Likelihood and Loss

Given observations x(1), ..., x(n), maximum likelihood estimation chooses parameters θ that make the observed data probable:

L(θ) = ∏i pθ(x(i)),

or, more conveniently, maximises the log-likelihood:

log L(θ) = Σi log pθ(x(i)).

In practice, optimisation commonly minimises negative log-likelihood. For language models this becomes cross-entropy loss over the next-token predictions. Lower loss indicates that the model assigns greater probability, on average, to the observed target tokens. It does not guarantee factual correctness, originality, social acceptability, or usefulness in a particular application.

Perplexity is a transformation of average negative log-likelihood often used for language models. It can help compare predictive performance under controlled conditions, but it is not a complete measure of generation quality. A model may have lower perplexity yet produce less reliable answers, because quality also depends on grounding, reasoning, calibration, safety, and user requirements.

Sampling and Uncertainty

After training, generation requires a sampling rule. Greedy decoding chooses the most probable next token at each step. It is computationally simple but can produce repetitive or locally plausible text. Beam search maintains several candidate sequences and is useful in some structured generation tasks, although it may favour generic outputs.

Temperature modifies the sharpness of a probability distribution. A lower temperature makes high-probability choices more dominant; a higher temperature spreads probability across more alternatives. Top-k sampling restricts choices to the k most probable tokens, whereas nucleus or top-p sampling chooses the smallest set whose cumulative probability exceeds p. These controls change the diversity and risk profile of outputs; they do not add knowledge to the model.

Uncertainty has several forms. Aleatoric uncertainty arises from inherent variability in the task or data. Epistemic uncertainty reflects incomplete knowledge or limited representation in the training data. In generative AI, uncertainty may appear as alternative valid continuations, ambiguous prompts, distribution shift, or unsupported claims. A model can express confidence through highly probable language while still being wrong. Calibrating confidence and designing mechanisms for abstention or verification are therefore important deployment concerns.

Latent Representations: Learning Hidden Structure

A latent variable is an unobserved variable used to explain regularities in observed data. In image generation, a latent vector may encode abstract factors such as pose, lighting, and object identity. In language, learned representations can capture syntactic, semantic, positional, and contextual relationships without assigning each dimension a simple human-readable meaning.

Latent representations reduce the burden of modelling raw observations directly. An image contains millions of pixel values, while a useful latent representation may describe a lower-dimensional structure from which images can be reconstructed. The representation is not necessarily a single interpretable concept; its dimensions may be distributed and entangled.

The latent-space perspective supports interpolation and conditional generation. If two latent vectors correspond to related examples, moving between them may produce a sequence of plausible intermediate outputs. However, smoothness and interpretability are properties to be evaluated, not assumptions to be made. A latent space can contain artefacts, demographic biases, or regions poorly supported by training data.

Major Generative Model Families

Autoregressive Models

An autoregressive model factorises a joint distribution into a sequence of conditional distributions. For text, the model predicts the next token given previous tokens. Training usually uses teacher forcing: the correct preceding tokens are supplied while the model learns to predict the next one. At inference time, however, the model consumes its own generated tokens. This difference creates exposure bias and can allow early errors to influence later output.

Autoregressive models are attractive because their likelihood objective is explicit and their factorisation is mathematically clear. They are particularly effective for ordered data such as text, music, code, and some forms of time series. Their principal limitation is sequential generation, which can be slower than highly parallel alternatives. They may also reproduce statistical associations without possessing reliable causal or factual understanding.

Large language models are predominantly autoregressive models. Given a prompt, they estimate a distribution over the next token. Repeated prediction generates an answer, program, summary, or dialogue turn. Their apparent intelligence results from large-scale parameterisation, extensive training, architectural improvements, and exposure to diverse patterns—not from a guarantee that every output has been logically verified.

Variational Autoencoders

A conventional autoencoder contains an encoder that maps an observation x to a latent representation and a decoder that reconstructs x. A variational autoencoder (VAE) introduces a probabilistic encoder qφ(z|x), a prior p(z), and a decoder pθ(x|z). Rather than mapping each input to one fixed point, the encoder produces a distribution over latent variables.

The VAE objective is commonly expressed through the evidence lower bound (ELBO):

ELBO = Eqφ(z|x)[log pθ(x|z)] − KL(qφ(z|x) || p(z)).

The reconstruction term encourages the decoded sample to resemble the input. The Kullback–Leibler divergence term encourages the learned posterior to remain close to a chosen prior, often a standard normal distribution. This regularisation makes it possible to sample a latent vector and decode it into a new observation.

VAEs provide structured and relatively smooth latent spaces, efficient inference, and a principled probabilistic objective. Their outputs can be blurrier than those of adversarial or diffusion models, especially when the likelihood model averages over multiple plausible details. They are useful in representation learning, anomaly detection, controlled generation, and situations where latent inference is important.

Generative Adversarial Networks

A generative adversarial network (GAN) consists of a generator and a discriminator. The generator maps random noise, and possibly conditioning information, to synthetic data. The discriminator attempts to distinguish real training examples from generated examples. The two networks are trained in opposition, commonly through a minimax objective.

The generator improves by producing samples that the discriminator finds difficult to reject. The discriminator improves by identifying differences between real and synthetic data. This adversarial process can produce sharp and visually compelling outputs without requiring an explicit tractable likelihood.

GAN training is difficult to stabilise. Mode collapse occurs when the generator produces limited varieties of outputs and fails to cover the full data distribution. Training may oscillate, and discriminator performance is not a simple measure of generator quality. GANs have been influential in image synthesis, super-resolution, style transfer, and data augmentation, but their use requires careful monitoring of diversity, fidelity, and memorisation.

Diffusion Models

Diffusion models generate data by learning to reverse a gradual noising process. In the forward process, noise is added to a clean sample through multiple steps until the signal is largely destroyed. A neural network is then trained to predict the noise, denoised sample, or a related parameterisation at each stage. Generation begins with random noise and iteratively removes noise according to the learned reverse process.

The approach provides stable training and high-quality, diverse samples. Conditioning can be supplied through text embeddings, class labels, masks, or other signals. Latent diffusion performs the process in a compressed latent space, reducing computation while retaining perceptual detail.

The main cost is iterative sampling. Although accelerated samplers and distillation methods reduce the number of steps, generation may remain computationally expensive. Diffusion systems can also inherit training-data bias, reproduce recognisable styles, generate unsafe content, or create images that appear plausible but contain structural errors.

Transformers as a General Architecture

The transformer is an architecture based primarily on attention mechanisms. Self-attention allows each position in a sequence to compute weighted relationships with other positions. For queries Q, keys K, and values V, scaled dot-product attention is commonly written as:

Attention(Q,K,V) = softmax(QKᵀ/√dk)V.

Multi-head attention uses several attention projections so that different relational patterns can be represented in parallel. Positional information is added because attention alone does not inherently encode sequence order. Feed-forward layers, residual connections, normalisation, and stacked blocks produce a deep contextual representation.

Transformers can be configured in several ways. Encoder-only models are effective for representation and classification tasks. Decoder-only models generate autoregressively and are widely used for large language models. Encoder–decoder models encode an input sequence and generate an output sequence, making them suitable for translation and text transformation.

Transformers are not a separate probabilistic objective from autoregression; rather, they are an architecture commonly used to implement it. The distinction matters. Autoregression describes how a distribution is factorised and generated, whereas the transformer describes how contextual computations are performed. Transformers have also been adapted for images, audio, video, and multimodal inputs.

Comparing the Families

Model selection should follow the structure of the problem rather than fashion. Autoregressive models are natural when order and conditional continuation are central. VAEs are suitable when a compact latent representation, approximate inference, and controllable sampling are important. GANs can provide sharp outputs but demand careful stability and diversity checks. Diffusion models offer strong sample quality and conditioning at the cost of iterative computation. Transformers provide flexible context modelling and scale effectively with data and compute, but their resource requirements and opacity can be substantial.

Important comparison criteria include the quality and diversity of samples, training stability, inference latency, likelihood availability, controllability, data efficiency, interpretability, privacy risk, and total cost of ownership. A model that performs well on a benchmark may be inappropriate for an organisation if it cannot meet latency, auditability, data-residency, or safety requirements.

Practical and Professional Applications

Generative models are used in content drafting, code assistance, document transformation, conversational interfaces, synthetic data generation, design exploration, education, drug discovery, media production, and simulation. In organisations, value generally depends on workflow integration rather than generation alone. A language model may reduce the time required to prepare a first draft, but human review, source verification, access controls, and records management remain necessary.

A practical deployment pipeline normally includes data preparation, model selection, conditioning or retrieval design, inference controls, evaluation, monitoring, and governance. Evaluation should combine automated metrics with expert judgement and task-level outcomes. For example, a customer-support assistant should be assessed not only for linguistic fluency but also for resolution accuracy, escalation behaviour, privacy protection, response time, and customer impact.

Synthetic data can help when real data are scarce or sensitive, but synthetic records do not automatically preserve rare cases or eliminate privacy risk. If a generator memorises training examples, its outputs may disclose information. Privacy assessment should therefore include memorisation testing, access restrictions, data minimisation, and, where appropriate, formal privacy techniques.

Illustrative Case Study: A Generative Support Assistant

Background

A fictional financial-services organisation, Northstar Mutual, receives a large volume of customer questions about account procedures, fees, and documentation. It proposes a generative assistant to support service agents rather than replacing them.

Situation and Problem

The organisation needs faster response drafting, but its policy documents change frequently. A general language model may produce fluent answers that conflict with current policy or disclose information from one customer to another. The central problem is therefore not merely text generation; it is controlled generation under factual, privacy, and regulatory constraints.

Analysis

The team identifies two distinct requirements. First, the system must retrieve authoritative policy passages relevant to a question. Second, it must generate a response grounded in those passages and defer when evidence is insufficient. A decoder-only transformer is selected for generation, while a retrieval component supplies current documents. Generation parameters are kept relatively conservative to reduce unnecessary variation.

Evaluation sets include ordinary questions, ambiguous requests, outdated-policy traps, attempts to obtain another person’s information, and questions outside the system’s authority. Reviewers score factual support, completeness, tone, privacy compliance, and appropriate escalation.

Approach and Outcome

The assistant is introduced in a staged pilot. It presents a draft and cited source passages to an agent, who remains responsible for sending the final response. The system refuses or escalates requests when retrieved evidence is absent or when identity and authorisation cannot be established. Monitoring tracks unsupported claims, escalation rates, latency, and changes in policy-document coverage.

The pilot demonstrates that the greatest benefit comes from reducing drafting effort while preserving human accountability. It also shows that retrieval quality and document governance are as important as model size. The assistant is not approved for autonomous financial decisions.

Learning Points

1. Generative quality must be defined in relation to the task and risk.

2. Current, authoritative context can be more valuable than unconstrained model expansion.

3. Human review is a control mechanism, not merely a temporary inconvenience.

4. Evaluation must include adversarial and out-of-scope cases.

5. Governance, identity management, logging, and document versioning are part of the AI system.

Challenges, Risks, and Common Mistakes

Generative systems may hallucinate: they can produce statements that are linguistically convincing but unsupported. They may reflect historical bias, amplify stereotypes, expose private information, infringe intellectual-property rights, or facilitate fraud and impersonation. Model outputs can also be brittle under prompt changes and may fail when deployment data differ from training data.

Common technical mistakes include evaluating only average benchmark performance, ignoring rare but harmful failures, treating confidence-like language as calibrated probability, and changing sampling settings without retesting safety. Organisations also underestimate operational costs such as inference infrastructure, monitoring, data curation, model updates, and incident response.

A robust approach includes representative evaluation data, red-team testing, provenance and citation where appropriate, content and access controls, human escalation, logging, version management, and continuous monitoring. Safety filters should not be treated as a complete solution; they must be combined with system design, policy, training, and accountability.

Ethical, Governance, and Professional Considerations

Generative AI raises questions about consent, copyright, authorship, labour, representation, accountability, and unequal impact. Organisations should establish who is responsible for an output, which data may be used, how users are informed, and when human approval is mandatory. High-impact applications require stronger validation and oversight than low-risk creative assistance.

Professional practice requires transparency about the role of AI, protection of confidential information, documentation of model limitations, and mechanisms for correction and appeal. Regulatory obligations vary by jurisdiction and application, but principles such as risk classification, data protection, traceability, security, and meaningful human oversight are broadly relevant.

Current Developments and Best Practices

Current research is moving toward multimodal models, efficient adaptation, retrieval-augmented generation, tool-using agents, smaller specialised models, synthetic-data pipelines, and improved evaluation of factuality and reasoning. Diffusion methods are being extended to video, three-dimensional assets, and scientific structures. Transformer research increasingly addresses long contexts, sparse computation, memory, and energy efficiency.

Best practice is to begin with a clearly defined task and failure tolerance. Select the simplest model that meets the requirement, establish a reliable evaluation set before deployment, separate generation from verification when possible, restrict access to sensitive data, and monitor real-world outcomes. Model choice should be justified by evidence, cost, controllability, and risk rather than by parameter count alone.

Chapter Summary

Generative AI models probability distributions or useful approximations to them so that new outputs can be sampled. Generative learning differs from discriminative learning because it focuses on modelling data or joint distributions rather than only predicting labels. Likelihood and cross-entropy provide important training objectives, while sampling strategies determine the diversity and predictability of outputs. Latent representations capture hidden structure and support reconstruction, interpolation, and conditional generation.

Autoregressive models factorise sequences into conditional predictions; VAEs learn probabilistic latent spaces through reconstruction and regularisation; GANs use competition between generators and discriminators; diffusion models learn to reverse a noise process; and transformers provide an attention-based architecture widely used for autoregressive language modelling and multimodal systems. Each family involves trade-offs in quality, diversity, speed, stability, controllability, and interpretability. Successful professional use requires evaluation, grounding, privacy protection, human oversight, and governance in addition to model training.

Key Terms

Generative learning: Learning a data or joint probability distribution in order to model and produce observations.

Discriminative learning: Learning a conditional distribution or decision boundary for predicting labels from inputs.

Probability distribution: A mathematical description assigning probabilities or densities to possible outcomes.

Likelihood: The probability assigned by a parameterised model to observed data, considered as a function of the parameters.

Log-likelihood: The logarithm of likelihood, used because products become sums and optimisation is more convenient.

Sampling: Drawing an output from a modelled probability distribution.

Uncertainty: Lack of certainty about an output, caused by variability, ambiguity, incomplete knowledge, or distribution shift.

Latent variable: An unobserved variable used to represent hidden structure in observed data.

Autoregressive model: A model that factorises a sequence distribution into conditional predictions of successive elements.

Variational autoencoder: A probabilistic encoder–decoder model trained using reconstruction and KL-divergence regularisation.

ELBO: Evidence lower bound used to optimise a variational approximation to data likelihood.

GAN: A generative adversarial network consisting of a generator and discriminator trained in competition.

Mode collapse: A GAN failure in which generated samples lack diversity and represent only limited modes.

Diffusion model: A model that learns to reverse a gradual noising process to generate data.

Transformer: An attention-based neural architecture for modelling relationships among sequence elements.

Self-attention: A mechanism that computes contextual relationships among positions within the same input.

Temperature: A decoding parameter controlling the concentration or spread of token probabilities.

Top-p sampling: Sampling restricted to the smallest set of tokens whose cumulative probability reaches p.

Hallucination: A plausible-looking but unsupported or incorrect generated statement.

Calibration: The degree to which stated or implied confidence corresponds to actual correctness probabilities.

Review Questions

Short Answer Questions

1. What is the principal difference between p(x) and p(y|x) in generative and discriminative modelling?

2. Why is log-likelihood generally preferred to likelihood for optimisation?

3. What role does a latent variable play in a VAE?

4. How does temperature affect autoregressive text generation?

5. Why is a transformer architecture not identical to an autoregressive objective?

Long Answer Questions

1. Explain the chain-rule factorisation used by autoregressive models and discuss its training and inference implications.

2. Describe the VAE ELBO objective and explain the roles of reconstruction and KL-divergence terms.

3. Compare GANs and diffusion models in terms of training process, sample quality, diversity, and computational cost.

4. Explain how self-attention operates and why transformers have become important for large language models.

5. Discuss why likelihood, perplexity, and fluency are insufficient as standalone measures of generative-system quality.

Analytical / Application Questions

1. A company needs a system that generates policy answers using frequently changing internal documents. Recommend an architecture and governance approach, and justify your choices.

2. An image generator produces highly realistic pictures but repeatedly depicts only a narrow range of people and settings. Identify possible causes and propose an evaluation and mitigation plan.

3. A language model gives confident answers to questions outside its training distribution. Analyse the uncertainty problem and design controls involving prompting, decoding, retrieval, verification, and human escalation.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes a generative model from a purely discriminative model?

A. A generative model can only classify examples into predefined labels.

B. A generative model attempts to represent how observations are distributed and can sample new observations.

C. A generative model never uses conditional probabilities.

D. A generative model does not require data for training.

Correct Answer: B. A generative model attempts to represent how observations are distributed and can sample new observations.

Question 2

For an autoregressive language model, which factorisation is represented by the chain rule?

A. p(x1,...,xT) = p(x1) multiplied by successive conditional token probabilities

B. p(x1,...,xT) = p(x1) + p(x2) + ... + p(xT)

C. p(x1,...,xT) is always uniform across all sequences

D. p(x1,...,xT) depends only on the final token

Correct Answer: A. p(x1,...,xT) = p(x1) multiplied by successive conditional token probabilities

Question 3

What is the main purpose of the KL-divergence term in a standard VAE objective?

A. To force every reconstruction to be identical to the input

B. To make the discriminator stronger than the generator

C. To regularise the approximate latent distribution toward a chosen prior

D. To remove the decoder from the model

Correct Answer: C. To regularise the approximate latent distribution toward a chosen prior

Question 4

Which problem is particularly associated with GAN training?

A. Mode collapse, in which generated samples lack diversity

B. The inability to use a discriminator

C. The requirement that all data be sequential text

D. The guaranteed production of blurry images

Correct Answer: A. Mode collapse, in which generated samples lack diversity

Question 5

How does a diffusion model typically generate an image?

A. By selecting the most common training image without modification

B. By reversing a learned gradual noising process, usually through multiple denoising steps

C. By classifying each pixel independently with no conditioning

D. By using only a fixed lookup table

Correct Answer: B. By reversing a learned gradual noising process, usually through multiple denoising steps

Question 6

Which statement correctly relates transformers and autoregressive modelling?

A. They are identical terms describing the same concept.

B. Autoregression is an architecture, whereas a transformer is only a loss function.

C. A transformer is an architecture that can implement an autoregressive objective, among other configurations.

D. Transformers cannot model sequences.

Correct Answer: C. A transformer is an architecture that can implement an autoregressive objective, among other configurations.

Question 7

A team lowers temperature during text generation. What is the most likely immediate effect?

A. The probability distribution becomes more concentrated around high-probability choices.

B. The model gains new factual knowledge.

C. The model is forced to retrieve external documents.

D. The model changes from generative to discriminative learning.

Correct Answer: A. The probability distribution becomes more concentrated around high-probability choices.

Question 8

Why might low perplexity fail to guarantee a reliable production assistant?

A. Perplexity measures only hardware speed.

B. Perplexity directly measures privacy compliance and factual accuracy.

C. Perplexity is unrelated to token prediction.

D. A model can predict likely text while still producing unsupported, unsafe, or contextually inappropriate answers.

Correct Answer: D. A model can predict likely text while still producing unsupported, unsafe, or contextually inappropriate answers.

Question 9

In the Northstar Mutual case, why was retrieval combined with generation?

A. To replace all policy documents with random samples

B. To provide current authoritative context and reduce unsupported answers

C. To ensure that the model never needs evaluation

D. To make human review impossible

Correct Answer: B. To provide current authoritative context and reduce unsupported answers

Question 10

Which practice is most appropriate for a high-risk generative AI deployment?

A. Evaluate only average benchmark scores before launch.

B. Treat confident wording as proof of correctness.

C. Use representative and adversarial testing, monitoring, access controls, and human escalation.

D. Increase temperature whenever the system makes an error.

Correct Answer: C. Use representative and adversarial testing, monitoring, access controls, and human escalation.