Foundations of Generative AI

Introduction and Context

Generative artificial intelligence (AI) refers to computational systems that produce new content—such as text, images, audio, video, software code, and synthetic data—in response to instructions or other inputs. Unlike traditional predictive analytics, which commonly estimates a label, probability, or numerical value, generative AI models a distribution of possible outputs and samples or decodes from that distribution.

For business analytics students, generative AI is important for two reasons. First, it is becoming an interface for analysing enterprise information, communicating insights, and automating knowledge work. Second, its outputs are probabilistic rather than guaranteed to be true. Effective use therefore requires both technical understanding and professional judgement.

Modern generative AI is built from several connected ideas: tokens represent input units; embeddings represent tokens as vectors; attention determines which contextual relationships matter; transformer networks process sequences; foundation models are pretrained on broad data; fine-tuning adapts models to particular tasks; inference generates outputs; and diffusion models generate images and other continuous data. Multimodal systems extend these capabilities across text, vision, audio, and video.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain foundation models, large language models, tokens, embeddings, and attention.
2. Describe the architecture and operating logic of transformer models.
3. Distinguish pretraining, fine-tuning, prompting, retrieval augmentation, and inference.
4. Explain how diffusion models generate images and other continuous media.
5. Analyse multimodal generation and its business applications.
6. Evaluate the limitations, risks, governance requirements, and professional implications of generative AI.

From Task-Specific Models to Foundation Models

Earlier machine-learning systems were usually designed for a defined task: classifying transactions as fraudulent, forecasting demand, or predicting customer churn. Such models typically required labelled examples that matched the target task. A foundation model is instead trained on broad and diverse data at substantial scale, then adapted to many downstream tasks.

The term “foundation” describes a model’s reusable base, not a guarantee of quality. A foundation model may support summarisation, classification, question answering, translation, coding, image generation, or data extraction. Its flexibility comes from learning general patterns in language, images, audio, or other data. However, broad training does not ensure current information, factual accuracy, fairness, or suitability for a specific organisation.

A foundation model normally has three lifecycle stages:

The model’s parameters—numerical values learned during training—encode statistical regularities. They do not function as a conventional database containing perfectly retrievable documents. Consequently, a model can produce fluent but unsupported statements, often called hallucinations.

Tokens and Representations

A language model does not directly read words in the human sense. Text is converted into tokens. A token may be a complete word, part of a word, punctuation, or a special control symbol. Tokenisation allows a model to represent unfamiliar words through smaller pieces, but it also means that token counts vary across languages, writing styles, and vocabulary.

A sequence of tokens can be represented as x₁, x₂, …, x_n, where n is the number of tokens in the input. The model has a context window: the maximum number of tokens it can process in one interaction. A context window is not the same as long-term memory. Information outside the current context is generally unavailable unless it is stored and retrieved by another system.

Each token is mapped to an embedding, a numerical vector. If the embedding dimension is d, token x_i may be represented by e_i ∈ ℝ^d. Similar usage patterns often produce vectors with related geometric relationships, although “similarity” depends on the training objective and context.

A common similarity measure is cosine similarity:

Here, a and b are vectors, a · b is their dot product, and ||a|| and ||b|| are their Euclidean lengths. The result is dimensionless and generally lies between −1 and 1 for non-zero vectors. Cosine similarity compares orientation rather than magnitude and is frequently used in semantic search. It does not prove that two documents have identical meaning.

Embeddings are useful in business analytics because they allow unstructured content to be compared, clustered, searched, or classified. For example, customer complaints can be embedded and grouped by semantic topic even when they use different vocabulary. A retrieval system can use embeddings to identify relevant policy documents before passing them to a language model.

Attention: Selecting Relevant Context

In a long sentence, the meaning of one token may depend on several other tokens. Attention enables a model to assign different weights to contextual elements. The standard scaled dot-product attention operation is:

Q is the query matrix, K is the key matrix, V is the value matrix, and d_k is the dimensionality of the key vectors. The transpose Kᵀ permits pairwise dot products. Division by √d_k controls the scale of scores, while softmax converts scores into non-negative weights that sum to 1 across the attended positions. This formula assumes compatible matrix dimensions.

Intuitively, a query asks, “Which information is relevant to me?” Keys describe what each position offers, and values contain the information passed forward. In self-attention, queries, keys, and values are derived from the same sequence. In cross-attention, they may come from different sequences—for example, text instructions attending to image features.

Transformers use multi-head attention. Instead of one attention operation, several attention heads learn different relationships, such as grammatical agreement, references across a paragraph, or relationships between an image region and a phrase. The outputs are combined and transformed through feed-forward layers.

Attention is powerful but computationally demanding. For a sequence of length n, pairwise attention commonly has time and memory requirements that grow approximately with n² for the attention matrix. This creates practical pressure to use efficient attention methods, summarisation, retrieval, or extended-context architectures.

Transformer Architecture

The transformer, introduced for sequence modelling without relying on recurrent processing, became the dominant architecture for many generative systems. A simplified transformer block contains:

1. Input representations, including token and positional information.
2. Self-attention, which integrates contextual relationships.
3. A residual connection and normalisation.
4. A position-wise feed-forward network.
5. Further residual connection and normalisation.

Because self-attention alone does not inherently identify order, models require positional information. Positional encodings or learned position embeddings provide information about token sequence order. Causal language models apply a mask so a position cannot attend to future tokens during next-token prediction.

A decoder-style language model estimates the probability of a token given preceding tokens. For a sequence x₁, …, x_T, the chain rule gives:

Here, T is sequence length and each conditional probability is produced by the model. In causal pretraining, the model learns by predicting the next token from preceding context. The product expresses a probability distribution over the entire sequence; in practice, logarithms are used for numerical stability.

An encoder-style transformer is commonly used to create contextual representations, while an encoder–decoder architecture can transform one sequence into another, such as translation. Decoder-only architectures are widely used for conversational language generation.

Large Language Models

A large language model (LLM) is a language model with substantial parameter count, training data, and computational scale. “Large” has no universal threshold. An LLM typically performs next-token prediction during pretraining and may subsequently receive instruction tuning and preference alignment.

Training commonly minimises cross-entropy loss. For a target token y_t and predicted probability p_θ(y_t | context_t), the average loss over N training positions can be written as:

θ denotes model parameters. N is the number of evaluated target positions. The logarithm is natural unless otherwise stated. Lower cross-entropy indicates better predictive performance on the evaluated data, but it does not by itself measure factuality, usefulness, fairness, or business value.

Perplexity is sometimes defined as:

when L is the average natural-log loss. Perplexity is a model-comparison metric under controlled conditions; it should not be interpreted as a direct percentage of correctness.

After pretraining, instruction tuning uses examples of instructions and desired responses. Preference optimisation or human-feedback methods may further encourage helpful, safe, and stylistically appropriate behaviour. These stages influence behaviour but do not eliminate bias or hallucination.

Pretraining, Fine-Tuning, Retrieval, and Inference

Pretraining

Pretraining exposes a model to large datasets and updates parameters through optimisation. Data preparation includes collection, deduplication, filtering, quality assessment, and—in responsible settings—licensing and privacy review. Training data may contain personal information, copyrighted material, stereotypes, or inaccurate content. The quality and governance of the dataset strongly influence model behaviour.

Fine-Tuning

Fine-tuning continues training from an existing model using task- or domain-specific examples. It can improve terminology, output format, classification performance, or organisational style. Full fine-tuning updates many or all parameters and can be expensive. Parameter-efficient methods update a smaller set of parameters or add trainable components, reducing computational and operational cost.

Fine-tuning is appropriate when the desired behaviour is stable and repeatedly expressed in examples. It is less appropriate for frequently changing facts, which are often better supplied through retrieval or controlled databases. Fine-tuning can also cause overfitting, reduce general capability, or encode poor practices if the examples are biased or inconsistent.

Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) combines information retrieval with generation. Documents are divided into passages, embedded, indexed, and retrieved in response to a query. Retrieved evidence is then placed in the model’s context. RAG can improve access to current internal information and support citations, but it depends on document quality, chunking, indexing, access control, and retrieval accuracy.

RAG is not a substitute for authoritative data governance. If the wrong document is retrieved, the model may produce a confident answer based on irrelevant evidence. Permission-aware retrieval is essential when enterprise documents contain confidential or personal data.

Inference and Decoding

Inference is the process of applying a trained model to an input. In a language model, generation proceeds token by token until a stopping condition is met. Decoding strategies affect output. Greedy decoding selects the highest-probability token at each step. Sampling introduces controlled randomness. Temperature modifies the concentration of probabilities: lower values generally produce more predictable output, while higher values increase variety but may reduce reliability. Top-k and nucleus (top-p) sampling restrict candidate tokens according to probability rules.

These controls do not create knowledge. They alter selection behaviour. For high-stakes tasks, organisations should use constrained formats, validation rules, retrieval, human review, and deterministic or low-variance settings where appropriate.

Diffusion Models and Generative Vision

Diffusion models are widely used for image, audio, and video generation. Their central idea is to learn how to reverse a gradual corruption process. During training, noise is progressively added to data. The model learns to estimate the noise or denoising direction at different time steps. During generation, the process begins with random noise and iteratively removes noise, guided by a text prompt, class label, or other condition.

A simplified forward noising relationship is:

Here, x₀ is the original data, x_t is the noisy version at time step t, ε is standard Gaussian noise, ᾱ_t is a schedule-dependent cumulative signal coefficient, and 0 ≤ ᾱ_t ≤ 1. The terms under the square roots are dimensionless coefficients, so x₀ and x_t have the same units or representation scale. The exact training and sampling equations vary across diffusion implementations.

Text-to-image systems commonly encode a prompt into a representation that conditions the denoising network through cross-attention. Latent diffusion performs the process in a compressed latent space rather than directly on every image pixel, improving computational efficiency. The generated image is then decoded into visible pixel space.

Diffusion models can support product concept visualisation, advertising drafts, architectural ideation, synthetic training data, and design exploration. They may also reproduce training-data styles, generate misleading media, or produce inaccurate text and objects. Human review remains important, especially for brand, safety, and legal requirements.

Multimodal Generation

Multimodal models process or generate more than one data type. A system may accept text and images, answer questions about charts, create an image from a description, transcribe audio, or produce a video from a storyboard. Modality-specific encoders convert inputs into representations that can be aligned or jointly processed.

For business analytics, multimodality enables:

• extracting information from invoices, forms, and charts;
• explaining a dashboard using natural language;
• analysing product images alongside sales data;
• generating voice or video explanations;
• supporting accessibility through speech, text, and visual conversion.

Multimodal systems introduce additional uncertainty. An image may be misread, a chart may be interpreted without its scale, or an audio transcription may omit negation. A robust workflow preserves the original source, records model outputs, checks confidence where available, and routes material decisions to qualified reviewers.

Worked Example: Semantic Search for Customer Complaints

Suppose an organisation has 10,000 customer complaints and wants to find cases semantically related to the query “delayed refund after a cancelled subscription.” A keyword search may miss a complaint stating, “The membership ended, but my money has not been returned.”

First, the organisation removes unnecessary formatting while preserving relevant text. Each complaint and the query are converted into embeddings. Cosine similarity is calculated between the query vector and each complaint vector. The highest-scoring passages are retrieved and presented to an analyst or language model for summarisation.

Assume the query vector is a = (0.8, 0.6) and a complaint vector is b = (0.6, 0.8). Their dot product is 0.8×0.6 + 0.6×0.8 = 0.96. Both vector lengths are 1, so:

The similarity is high under this simplified example. In a real system, embeddings have many dimensions, and a similarity score is only a ranking signal. The organisation should evaluate retrieval using labelled queries, inspect false positives and false negatives, protect customer data, and ensure that generated summaries remain traceable to source complaints.

Illustrative Case: Generative AI for a Retail Planning Team

Background

A fictional national retailer receives weekly sales reports, supplier messages, store-manager comments, and inventory spreadsheets. Planning analysts spend considerable time combining these sources into a management briefing.

Problem

The retailer wants faster reporting but cannot allow an AI system to invent sales figures, expose supplier-confidential information, or make unauthorised replenishment decisions.

Analysis

The information is heterogeneous and changes frequently. Fine-tuning alone would not ensure access to the latest figures. The project team therefore selects a retrieval-based architecture. Structured sales data remain in governed analytics systems; documents are indexed for semantic retrieval; and the language model is instructed to cite retrieved evidence and distinguish facts from recommendations.

Approach and Solution

The workflow includes:

1. Role-based access controls for sales, supplier, and personnel information.
2. Data-quality checks before reports become searchable.
3. Embeddings for document retrieval and metadata filters for region and date.
4. A language model that generates a draft briefing from retrieved passages and validated metrics.
5. Automated checks for missing citations, unsupported numerical claims, and prohibited disclosures.
6. Human approval by a planning manager before distribution.
7. Logging of prompts, retrieved sources, outputs, corrections, and model versions.

Outcome

In this illustrative case, drafting time falls substantially, while final accountability remains with the planning team. The system is used for summarisation and issue identification, not autonomous purchasing. Evaluation includes factual accuracy, citation coverage, retrieval recall, time saved, user acceptance, and incidents involving privacy or incorrect recommendations.

Learning Points

The case demonstrates that business value comes from integrating a model with reliable data, access controls, evaluation, and human processes. Selecting a larger model alone would not solve outdated data, ambiguous ownership, or weak governance.

Limitations, Risks, and Governance

Generative AI systems have several important limitations:

• Hallucination: fluent output may be factually unsupported.
• Bias: training data and design choices may reproduce or amplify unequal treatment.
• Distribution shift: performance may decline when real-world inputs differ from training data.
• Security threats: prompt injection, data exfiltration, model abuse, and insecure tool use can compromise systems.
• Privacy: prompts and outputs may contain personal, confidential, or commercially sensitive data.
• Copyright and provenance: training and generated content may create licensing, attribution, or ownership questions.
• Opacity: parameter-based systems are difficult to explain in the same manner as a simple decision tree.
• Environmental and financial cost: training and inference consume computing resources and electricity.
• Automation bias: users may accept an authoritative-sounding answer without checking it.

Good governance connects technical controls with organisational accountability. It should define approved use cases, data classifications, model ownership, evaluation criteria, incident reporting, retention rules, vendor obligations, and human-oversight requirements. High-impact decisions such as employment, credit, healthcare, or public services may require additional legal and ethical safeguards depending on jurisdiction.

A practical evaluation framework should test:

1. Capability: Can the system perform the intended task?
2. Reliability: Does performance remain stable across realistic inputs?
3. Grounding: Are claims supported by authoritative evidence?
4. Fairness: Are error rates and outcomes acceptable across relevant groups?
5. Security: Can malicious inputs or unauthorised users manipulate the system?
6. Usability: Can employees understand and appropriately challenge outputs?
7. Value: Do benefits exceed implementation, monitoring, and risk-management costs?

Best Practices for Business Analytics Professionals

Professionals should begin with a clearly defined business problem rather than with a model brand. They should identify the cost of errors, the data required, the decision owner, and the appropriate degree of automation. Inputs should be minimised, protected, and separated according to sensitivity.

Prompt design can improve clarity by specifying the task, audience, constraints, output format, evidence requirements, and uncertainty expectations. However, a well-written prompt cannot guarantee truth. Important outputs should be checked against source data, calculations, policies, or domain experts.

For quantitative work, language models should not be treated as calculators without verification. A safer architecture delegates arithmetic to validated software or a database and uses the model to explain results. For documents, citations and source links support auditability. For automated workflows, structured outputs, schema validation, rate limits, fallback procedures, and monitoring are essential.

Contemporary developments include longer-context models, smaller models deployed locally, multimodal agents, tool-using systems, synthetic data generation, model compression, and retrieval systems with improved ranking. These developments may reduce cost or expand capability, but they also increase the need for testing, access control, and clear responsibility boundaries.

Chapter Summary

Generative AI systems learn patterns from data and produce new outputs through probabilistic generation. Foundation models provide reusable capabilities across tasks, while large language models specialise in token-based language modelling. Tokens are converted into embeddings, and attention enables models to integrate contextual relationships. Transformers use attention, positional information, feed-forward layers, and residual connections to process sequences.

Pretraining establishes general capabilities; fine-tuning adapts behaviour to particular examples; retrieval-augmented generation supplies changing or private information; and inference applies the model to a live input. Decoding settings influence variability but do not guarantee accuracy. Diffusion models generate images and other media by learning to reverse a noise-adding process. Multimodal models combine text, visual, audio, and other representations.

For business analytics, generative AI can improve search, reporting, communication, design, and knowledge work. Its value depends on data quality, retrieval, evaluation, security, privacy, human oversight, and process integration. Hallucination, bias, privacy exposure, copyright uncertainty, prompt injection, and automation bias require systematic governance. A responsible professional treats generated content as an auditable, reviewable output—not as unquestionable fact.

Key Terms

Foundation model: A broadly pretrained model adaptable to multiple downstream tasks.

Generative AI: AI that creates new content or data-like outputs.

Token: A unit of input representation used by a language model.

Context window: The maximum token sequence a model can process in one interaction.

Embedding: A numerical vector representation of an item such as text or an image.

Cosine similarity: A measure comparing the orientation of two non-zero vectors.

Attention: A mechanism that weights contextual information according to relevance.

Self-attention: Attention in which queries, keys, and values derive from the same sequence.

Transformer: A neural architecture built around attention and feed-forward processing.

Large language model: A large-scale model trained to represent and generate language.

Pretraining: Initial broad training on extensive data.

Fine-tuning: Continued training for a particular domain, task, or behaviour.

Inference: Applying a trained model to generate or predict an output.

Retrieval-augmented generation: Combining information retrieval with model generation.

Diffusion model: A generative model that learns to reverse a gradual noising process.

Multimodal model: A model that processes or generates multiple data modalities.

Hallucination: An unsupported or incorrect output presented as if it were valid.

Prompt injection: A malicious or unintended instruction that manipulates an AI workflow.

Temperature: A decoding parameter influencing the concentration of token probabilities.

Human oversight: Review and accountability by qualified people in an AI-supported process.

Review Questions

5 Short Answer Questions

1. What is a foundation model, and why is it useful for multiple business tasks?
2. Explain the difference between a token and an embedding.
3. What role does attention play in a transformer?
4. Distinguish pretraining from fine-tuning.
5. Why might retrieval-augmented generation be preferable to fine-tuning for changing business facts?

5 Long Answer Questions

1. Describe the principal components of a transformer and explain how they support language generation.
2. Explain the scaled dot-product attention formula, defining its variables and discussing the purpose of scaling.
3. Compare large language models and diffusion models in terms of data representation, generation process, and business use.
4. Discuss the benefits and limitations of multimodal generative AI for business analytics.
5. Develop a governance framework for an organisation introducing a generative AI reporting assistant.

3 Analytical / Application Questions

1. A retailer wants an assistant that answers questions about current inventory, supplier contracts, and historical sales. Recommend an architecture and justify the roles of structured databases, retrieval, language models, access controls, and human review.
2. A generated management report contains accurate-looking but unsupported claims. Analyse possible technical and organisational causes, then propose evaluation and control measures.
3. Compare two proposed projects: automatic generation of marketing images and automated approval of consumer credit applications. Analyse how risk, oversight, data governance, and acceptable automation should differ between the projects.

Multiple Choice Questions with Answers

Question 1

Which statement best describes a foundation model?

A. A model trained only for one narrowly defined classification task

B. A broadly pretrained model that can be adapted to multiple tasks

C. A database containing every document used during training

D. A deterministic rules engine for business workflows

Correct Answer: B. A broadly pretrained model that can be adapted to multiple tasks

Question 2

What is the primary purpose of an embedding?

A. To represent an item as a numerical vector

B. To guarantee that generated text is factually correct

C. To remove all bias from a training dataset

D. To determine the legal ownership of generated content

Correct Answer: A. To represent an item as a numerical vector

Question 3

In scaled dot-product attention, why is QKᵀ divided by √d_k?

A. To convert tokens into words

B. To control the scale of attention scores

C. To eliminate the need for positional information

D. To ensure every token receives equal weight

Correct Answer: B. To control the scale of attention scores

Question 4

Which activity is characteristic of causal language-model pretraining?

A. Predicting a future token from preceding context

B. Adding random noise to an image and never removing it

C. Assigning employees to departments using fixed rules

D. Sorting documents alphabetically

Correct Answer: A. Predicting a future token from preceding context

Question 5

When is retrieval-augmented generation especially useful?

A. When information changes frequently and should be supplied from current sources

B. When an organisation wants to remove all source documents

C. When no access controls are required

D. When the model must operate without any external information

Correct Answer: A. When information changes frequently and should be supplied from current sources

Question 6

What is the main generation principle of a diffusion model?

A. It reverses a learned gradual noising process

B. It retrieves only exact copies of stored images

C. It predicts database rows using linear regression only

D. It converts every image directly into a single token

Correct Answer: A. It reverses a learned gradual noising process

Question 7

What does inference mean in the generative-AI lifecycle?

A. Collecting and licensing all possible training data

B. Applying a trained model to an input to produce an output

C. Deleting the model’s parameters after training

D. Manually labelling every possible future response

Correct Answer: B. Applying a trained model to an input to produce an output

Question 8

Which risk occurs when a model produces fluent but unsupported claims?

A. Hallucination

B. Tokenisation

C. Positional encoding

D. Quantisation error in a physical sensor

Correct Answer: A. Hallucination

Question 9

If two unit-length vectors have a dot product of 0.96, what is their cosine similarity?

A. 0.48

B. 0.92

C. 0.96

D. 1.96

Correct Answer: C. 0.96

Question 10

Which control most directly improves auditability of an AI-generated business briefing?

A. Increasing temperature substantially

B. Removing citations to make the report shorter

C. Logging sources, prompts, outputs, corrections, and model versions

D. Allowing the model to approve its own recommendations

Correct Answer: C. Logging sources, prompts, outputs, corrections, and model versions