Attention, Transformers and Foundation Models

Introduction

Deep learning systems initially processed sequences with recurrent or convolutional architectures. Recurrent neural networks (RNNs) imposed a sequential computation pattern, while convolutional models used a local receptive field. Both approaches made it difficult to model relationships between distant elements efficiently. Attention introduced a different principle: each element can selectively retrieve information from other elements according to learned relevance.

The transformer architecture, built primarily from attention and position representations, made sequence modelling highly parallelizable and scalable. Large pretrained transformers subsequently became foundation models: broadly trained models that can be adapted to many downstream tasks through prompting, supervised fine-tuning, parameter-efficient fine-tuning, or retrieval augmentation. Although language models popularized this paradigm, transformer-based methods now support vision, audio, video, robotics, scientific computing and multimodal reasoning.

This chapter develops the mathematical intuition and engineering practice of attention and transformers, then examines how pretrained models are adapted and governed in professional settings.

Learning Objectives

After completing this chapter, students should be able to:

1. Explain scaled dot-product attention and multi-head attention.
2. Describe the architecture and information flow of an encoder, decoder and encoder–decoder transformer.
3. Compare positional encoding methods and analyse computational complexity.
4. Explain how pretraining produces foundation models and how prompting differs from fine-tuning.
5. Select among full fine-tuning, parameter-efficient fine-tuning and retrieval-augmented generation (RAG).
6. Adapt language, vision and multimodal models while evaluating reliability, safety and governance risks.

1. From Sequence Models to Attention

For a sequence of vectors X = (x₁, …, xₙ), an RNN computes a hidden state sequentially. This creates a dependency between time steps and limits parallelism during training. Long-range information must also pass through many transformations. Attention removes the requirement that information travel through a single recurrent chain. A query can directly access all relevant keys and values.

Attention is therefore a content-addressable information-retrieval mechanism. Queries represent what a position is seeking, keys represent what each position offers for matching, and values contain the information retrieved after matching.

1.1 Scaled dot-product attention

Let Q ∈ ℝⁿˣᵈᑫ, K ∈ ℝᵐˣᵈᑫ and V ∈ ℝᵐˣᵈᵛ denote query, key and value matrices. The standard attention operation is:

Here, dₖ is the key dimension; QKᵀ contains pairwise compatibility scores; M is an optional mask; and softmax is applied row-wise. If a position is forbidden from attending to another position, the corresponding element of M is set to −∞ conceptually, or to a sufficiently large negative number in implementation.

The scaling factor √dₖ is important. If query and key components have approximately zero mean and unit variance, the variance of their dot product tends to grow with dₖ. Large logits make softmax highly peaked and produce small gradients. Division by √dₖ stabilizes the score distribution.

For one query q and key–value pairs (kⱼ, vⱼ), the output is a weighted sum:

The weights satisfy αⱼ ≥ 0 and ∑ⱼ αⱼ = 1, assuming an ordinary softmax over at least one unmasked key. Thus, each output is a convex combination of values, although subsequent linear layers can transform it further.

1.2 Self-attention and cross-attention

In self-attention, Q, K and V are learned projections of the same sequence. Each token can integrate contextual information from other tokens. In cross-attention, queries come from one sequence and keys and values from another. In an encoder–decoder translation model, decoder queries attend to encoder outputs.

Three masking patterns are especially important:

A causal mask prevents position i from using positions greater than i. Without it, a model trained to predict the next token could directly observe the answer during training.

1.3 Multi-head attention

A single attention operation may focus on only one type of relationship. Multi-head attention uses h parallel projections:

WᵢQ, WᵢK and WᵢV are learned projection matrices, and Wᴼ maps the concatenated outputs back to the model dimension. If the model dimension is dₘ and heads have equal size, each head commonly uses dₖ = dᵥ = dₘ/h. This keeps the total projection cost comparable to a single full-dimensional attention layer.

Different heads may learn different patterns, such as local syntactic relations, long-distance agreement or positional alignment. However, interpreting individual heads as fixed linguistic modules is not universally justified; head functions can be redundant, distributed and dependent on the surrounding network.

2. Transformer Architecture

A transformer layer generally combines attention, a position-wise feed-forward network, residual connections and normalization. A typical pre-layer-normalized block is:

LN denotes layer normalization. The feed-forward network is applied independently at each position:

W₁ and W₂ are learned matrices, b₁ and b₂ are biases, and σ is a nonlinear activation such as GELU or a gated variant. Attention mixes information across positions; the FFN transforms each position’s representation. Residual connections improve optimization by providing short paths for information and gradients.

2.1 Encoder, decoder and encoder–decoder designs

An encoder contains bidirectional self-attention and produces contextual representations. It is suitable for classification, retrieval and token-level labelling. A decoder contains causal self-attention and predicts a sequence autoregressively. An encoder–decoder model first encodes an input and then generates an output using causal self-attention plus cross-attention.

Autoregressive factorization expresses a sequence probability as:

The equality follows from the chain rule of probability. During training, teacher forcing supplies the true preceding tokens; during inference, previously generated tokens are fed back into the decoder.

2.2 Positional information

Attention alone is permutation-equivariant: rearranging inputs and rearranging outputs in the same way does not change the underlying operation. A model therefore needs positional information to distinguish “the dog bites the man” from “the man bites the dog.”

Common approaches include:

• Absolute learned embeddings: a learned vector is associated with each position up to a configured maximum.
• Fixed sinusoidal encodings: deterministic sine and cosine functions provide multiple frequency scales and can sometimes extrapolate beyond training lengths.
• Relative position methods: attention scores or representations depend on distance between positions.
• Rotary positional embeddings (RoPE): position-dependent rotations are applied to query and key components, making attention scores reflect relative displacement.
• Attention biases: a position-dependent term is added directly to attention logits.

No positional method guarantees reliable long-context extrapolation. The training length, interpolation strategy, numerical precision and task distribution all matter.

2.3 Complexity and memory

For sequence length n and model width d, forming the attention score matrix has O(n²d) computational complexity and O(n²) score-memory complexity. This quadratic dependence is a major constraint for long documents, images with many patches and video tokens. The projection and feed-forward costs are approximately linear in n for fixed d, but the attention matrix can dominate as n grows.

Efficient attention methods reduce memory, approximate the score matrix, restrict attention to local or block patterns, compress tokens, or use recurrent/state-space alternatives. These methods trade exact global interaction for lower cost and should be evaluated on the target context distribution rather than assumed to be universally superior.

3. Pretraining and Foundation Models

A foundation model is trained on broad data at scale and adapted to many tasks. The term describes a development paradigm rather than a single architecture. A model may be trained with language modelling, masked reconstruction, contrastive learning, image–text alignment, diffusion objectives or mixtures of objectives.

3.1 Language-model objectives

In causal language modelling, the model estimates the next-token distribution. Given logits z over vocabulary V, the softmax probability for token j is:

The token-level cross-entropy loss for a target sequence is:

T is the number of target tokens, y₍<ₜ₎ denotes preceding target tokens, and θ represents model parameters. The loss is measured in nats when natural logarithms are used. Perplexity is exp(L) under this normalization; it is a model-based uncertainty measure, not a direct measure of factual correctness or usefulness.

Masked language models hide selected tokens and predict them from both left and right context. This supports strong representations but does not directly define left-to-right generation. Sequence-to-sequence objectives train a model to reconstruct, transform or generate one sequence from another.

3.2 Vision and multimodal models

Vision transformers divide an image into patches, flatten each patch, project it into an embedding and process the resulting token sequence. A class token or pooled representation may support classification. The number of patches depends on image dimensions and patch size, so reducing patch size increases spatial detail but raises attention cost.

Multimodal systems connect modalities in several ways. A shared embedding space can align images and text through contrastive learning. A vision encoder can feed image tokens into a language decoder through a projection or cross-attention module. Unified models may tokenize text, image, audio and video into a common sequence, although tokenization and training efficiency remain difficult design issues.

4. Adapting Pretrained Models

Adaptation begins with a task specification, data audit and baseline. The practitioner should identify the required output, acceptable error types, latency and privacy constraints before choosing a method.

4.1 Prompting

Prompting changes the input or instruction while keeping model parameters fixed. Zero-shot prompting supplies a task description; few-shot prompting includes demonstrations. Structured prompts can specify role, context, constraints, output schema and refusal conditions. Prompt quality is not merely wording: demonstrations, ordering, delimiters and ambiguity affect performance.

For reliable applications, prompts should be versioned and tested against adversarial, ambiguous and out-of-distribution inputs. A prompt is not a security boundary. Instructions embedded in retrieved documents or user content may attempt prompt injection, so trusted instructions and untrusted data must be separated operationally.

Decoding also affects output. Temperature modifies logits before softmax; lower values generally make distributions more concentrated. Top-k sampling restricts choices to k tokens, while top-p sampling chooses the smallest candidate set whose cumulative probability reaches p. Greedy decoding selects the highest-probability token at each step but is not guaranteed to maximize the probability of the complete sequence.

4.2 Supervised fine-tuning

Full fine-tuning updates all or most model parameters using task examples. It can produce strong task specialization but requires substantial memory, careful learning-rate selection and protection against catastrophic forgetting. Data quality is usually more important than simply increasing the number of examples. Training, validation and test sets must be separated by entity or time where leakage is possible.

Instruction fine-tuning teaches a model to follow task descriptions and produce preferred response formats. Preference optimization methods use rankings or preference signals to adjust behaviour, but preference data can encode annotator, cultural and institutional biases.

4.3 Parameter-efficient fine-tuning

Parameter-efficient fine-tuning (PEFT) adapts a model while freezing most pretrained weights. Methods include adapters, prompt tuning, prefix tuning and low-rank adaptation (LoRA). In LoRA, an update to a weight matrix W is represented as:

A ∈ ℝʳˣᵈ and B ∈ ℝᵈ′ˣʳ are trainable low-rank matrices, r is the rank, and α is a scaling factor. The original W is frozen. If r is much smaller than min(d,d′), the trainable parameter count is greatly reduced. The exact parameter saving depends on which matrices are adapted and whether biases or normalization parameters are also trained.

PEFT reduces storage and often lowers optimization cost, but it does not remove the need for representative data, evaluation or safety controls. A small adapter can still encode confidential information or harmful behaviour.

4.4 Retrieval-augmented generation

RAG supplies a generator with evidence retrieved from an external collection. A basic pipeline is:

1. Ingest, clean and segment authoritative documents.
2. Compute embeddings and store them in a vector index, optionally with lexical indexes and metadata filters.
3. Retrieve candidates for a user query.
4. Rerank or filter candidates for relevance, permissions and freshness.
5. Insert evidence into a controlled generation prompt.
6. Produce an answer with citations and evaluate both retrieval and generation.

RAG is valuable when knowledge changes frequently or must remain traceable. It does not guarantee truth: poor chunking, missing documents, semantic mismatch, stale indexes or faulty synthesis can still produce incorrect answers. Access control must be applied before content reaches the model, not only after generation.

Useful retrieval metrics include recall at k, precision at k and mean reciprocal rank. If a relevant item appears at rank r for a query, reciprocal rank is 1/r; mean reciprocal rank averages this quantity across queries. Generation should additionally be tested for citation correctness, completeness, refusal behaviour and answer faithfulness.

5. A Practical Adaptation Framework

A defensible adaptation decision can follow this sequence:

The choice should be empirical. Establish a baseline, define acceptance thresholds, run ablations and measure cost per successful task rather than relying only on benchmark scores.

6. Illustrative Case: University Policy Assistant

Background

A university wishes to answer questions about assessment regulations, attendance policies and academic appeals. The documents change each semester and contain access-controlled procedures.

Problem

A general language model can produce fluent answers but may invent deadlines, confuse faculty-specific rules or disclose procedures not applicable to a student. The university requires citations, current information and an escalation path.

Analysis

Full fine-tuning on historical policy documents would not reliably solve freshness: the model would still require retraining after each revision. It could also memorize obsolete rules. A RAG system is more appropriate, supplemented by a small adapter only if the desired response format is difficult to achieve through prompting.

Approach

Documents are versioned, approved by policy owners and segmented with section identifiers and effective dates. Retrieval applies student role, campus and faculty permissions before ranking passages. The prompt requires the model to answer only from supplied evidence, cite section identifiers, state when evidence is insufficient and direct high-risk cases to an administrative officer. Evaluation includes ordinary questions, conflicting versions, prompt injection in documents, misspellings, multilingual queries and deliberately unanswerable questions.

Outcome

In this illustrative scenario, the system’s success criterion is not fluency alone. It must retrieve the current applicable rule, cite it accurately, avoid unsupported conclusions and preserve privacy. A dashboard monitors retrieval recall, citation validity, abstention rate, latency and user corrections. Policy owners retain authority to remove documents and audit changes.

Learning points

The case demonstrates that model adaptation is a system-design problem. Retrieval, authorization, document governance, prompting, evaluation and human escalation are jointly responsible for reliability. A larger model cannot compensate for an incorrect source corpus or an inadequate access-control design.

7. Evaluation, Risks and Governance

Evaluation should be task-specific and multidimensional. Classification tasks may use accuracy, macro-F1 or calibration; generation tasks require factuality, relevance, completeness, citation accuracy, toxicity and robustness measures. Human evaluation protocols should define rubrics, annotator training and disagreement handling. Automated judges can assist but should not be treated as neutral ground truth.

Important risks include:

• Hallucination: fluent unsupported claims.
• Bias: unequal performance or harmful stereotypes associated with data and deployment context.
• Privacy leakage: reproduction or inference of personal or confidential information.
• Copyright and provenance concerns: uncertain rights over training, retrieval and generated material.
• Prompt injection and tool misuse: untrusted content manipulating instructions or actions.
• Distribution shift: degraded performance on new domains, languages, populations or image conditions.
• Automation bias: users accepting model outputs without appropriate scrutiny.
• Environmental and financial cost: energy, hardware, inference and data-management expenses.

Governance should include data minimization, documented provenance, risk assessment, access control, model and prompt versioning, red-team testing, incident response, monitoring and meaningful human oversight. In regulated environments, records should support auditability and explain which model, data sources and policies influenced an output. Privacy laws, copyright rules, sectoral regulation and institutional policy must be considered for the deployment jurisdiction; technical capability does not establish legal permission.

8. Contemporary Directions

Current research investigates longer-context mechanisms, sparse and linear attention, mixture-of-experts models, multimodal reasoning, tool use, model compression and test-time adaptation. Retrieval is becoming more structured, incorporating knowledge graphs, query planning and agentic workflows. Small language models and specialized models can be preferable where latency, privacy or on-device operation matters.

Several research questions remain open: how to measure factuality independently of surface fluency; how to evaluate reasoning rather than memorized patterns; how to prevent representation and capability loss during adaptation; how to establish robust uncertainty estimates; and how to make multimodal models reliable under distribution shift. Benchmark performance should therefore be interpreted as evidence under particular data and protocols, not as a complete measure of general intelligence.

Best Practices

1. Define the task, users, risks and acceptance thresholds before selecting a model.
2. Use the smallest capable model and disclose meaningful limitations.
3. Keep trusted instructions separate from untrusted user and retrieved content.
4. Treat retrieval quality, authorization and document freshness as first-class engineering concerns.
5. Use held-out, adversarial and out-of-distribution evaluation sets.
6. Prefer PEFT when specialization is required but full retraining is unnecessary.
7. Monitor production drift, failures, cost, latency and user corrections.
8. Provide citations, abstention and human escalation for high-impact decisions.
9. Record model, data, prompt, adapter and index versions.
10. Never treat fluent output as evidence of correctness.

Chapter Summary

Attention computes content-dependent weighted combinations of values using queries and keys. Scaling, masking and multi-head projections make the operation numerically stable and expressive. Transformers combine attention with positional information, feed-forward networks, residual connections and normalization; encoder, decoder and encoder–decoder variants support different tasks. Their quadratic attention cost motivates efficient alternatives.

Foundation models are broadly pretrained systems adapted through prompting, supervised fine-tuning, PEFT or RAG. Prompting is inexpensive but may be unstable; fine-tuning changes model behaviour but requires careful data and evaluation; PEFT reduces trainable parameters; RAG connects generation to changing or private evidence but requires strong retrieval and authorization controls. Effective deployment is a socio-technical process involving evaluation, governance, privacy, security, provenance and human oversight.

Key Terms

Attention: A mechanism that weights information from multiple elements according to learned relevance.

Query: A representation used to request information in attention.

Key: A representation matched against a query.

Value: The information returned after attention weights are computed.

Self-attention: Attention in which queries, keys and values originate from the same sequence.

Cross-attention: Attention from one sequence to representations of another sequence.

Causal mask: A mask preventing a position from attending to future positions.

Multi-head attention: Parallel attention operations with separate learned projections.

Transformer: A neural architecture centred on attention, feed-forward layers and residual pathways.

Positional encoding: Information that represents token or patch order.

Foundation model: A broadly pretrained model adaptable to many downstream tasks.

Causal language model: A model trained to predict the next token from preceding context.

Fine-tuning: Updating pretrained parameters using task-specific data.

Parameter-efficient fine-tuning: Adaptation that trains only a small subset or additional parameter set.

LoRA: A low-rank parameter-update method for efficient adaptation.

Prompting: Conditioning a frozen model through instructions and examples.

Retrieval-augmented generation: Generation conditioned on externally retrieved evidence.

Hallucination: An unsupported or fabricated model output.

Prompt injection: An attempt by untrusted content to alter a model’s intended instructions.

Perplexity: The exponential of average token cross-entropy under a specified evaluation distribution.

Review Questions

Short Answer Questions

1. Why is the dot product in attention divided by √dₖ?
2. Distinguish self-attention from cross-attention.
3. What problem does a causal mask solve?
4. Explain one difference between prompting and fine-tuning.
5. Why does RAG not guarantee factual answers?

Long Answer Questions

1. Explain the architecture of a transformer layer, including attention, normalization, residual connections and the feed-forward network.
2. Compare encoder-only, decoder-only and encoder–decoder transformers, giving an appropriate application for each.
3. Discuss positional encoding methods and the consequences of omitting positional information.
4. Compare full fine-tuning, LoRA and prompting in terms of data, computation, flexibility and risks.
5. Design an evaluation and governance plan for a multimodal foundation-model application.

Analytical / Application Questions

1. A company has frequently changing internal policies and strict employee permissions. Decide whether prompting, fine-tuning, PEFT or RAG should be the primary approach, and justify the architecture and controls.
2. An attention system uses sequence length n = 4,000. Analyse why score-matrix memory may be problematic and propose two technically different mitigation strategies, including their trade-offs.
3. A vision–language assistant gives fluent but poorly cited answers when retrieved passages are outdated. Diagnose the likely system failures and propose an evaluation plan separating retrieval, authorization, generation and citation errors.

Multiple Choice Questions with Answers

Question 1

In scaled dot-product attention, what is the primary purpose of dividing scores by √dₖ?

A. To ensure all values are positive

B. To stabilize softmax gradients when key dimension is large

C. To remove the need for positional information

D. To make attention computation linear in sequence length

Correct Answer: B. To stabilize softmax gradients when key dimension is large

Question 2

Which attention pattern is required for autoregressive next-token generation?

A. Bidirectional self-attention without a mask

B. Causal self-attention

C. Cross-attention only

D. Pooling without attention

Correct Answer: B. Causal self-attention

Question 3

In a transformer, what primarily mixes information across sequence positions?

A. The position-wise feed-forward network

B. Layer normalization

C. Attention

D. The output bias

Correct Answer: C. Attention

Question 4

Which model design normally uses bidirectional self-attention for producing contextual representations?

A. Encoder-only transformer

B. Decoder-only transformer during causal generation

C. A retrieval index

D. A tokenizer

Correct Answer: A. Encoder-only transformer

Question 5

In LoRA, which expression represents the adapted weight matrix?

A. W′ = W − α

B. W′ = W + (α/r)BA

C. W′ = softmax(W)

D. W′ = WᵀW

Correct Answer: B. W′ = W + (α/r)BA

Question 6

Which situation most strongly motivates retrieval-augmented generation?

A. The task requires no external knowledge

B. The knowledge changes frequently and must be cited

C. The model must forget all context

D. Only image resolution needs to increase

Correct Answer: B. The knowledge changes frequently and must be cited

Question 7

What does a causal mask prevent?

A. A token from attending to its own embedding

B. A position from attending to future positions

C. The model from using residual connections

D. The tokenizer from splitting words

Correct Answer: B. A position from attending to future positions

Question 8

If average token cross-entropy is L using natural logarithms, perplexity is defined as:

A. L²

B. 1/L

C. exp(L)

D. log(L)

Correct Answer: C. exp(L)

Question 9

Which is a security risk specific to untrusted retrieved or user-provided content?

A. Layer normalization

B. Prompt injection

C. Patch embedding

D. Residual addition

Correct Answer: B. Prompt injection

Question 10

Why can smaller patch sizes increase the computational burden of a vision transformer?

A. They reduce the number of image tokens

B. They increase the number of image tokens and potentially the quadratic attention cost

C. They eliminate positional information

D. They prevent image tokenization

Correct Answer: B. They increase the number of image tokens and potentially the quadratic attention cost