Attention, Transformers and Generative AI

Introduction

Modern natural-language processing (NLP) changed substantially when models began to represent relationships between all tokens in a sequence rather than processing words only from left to right. The attention mechanism made this possible by allowing a model to assign different importance to different parts of an input. Transformers built attention into a highly parallel architecture and became the foundation of systems for translation, summarisation, question answering, code generation, image understanding and multimodal interaction.

This chapter explains attention and transformer architectures, then connects them to foundation models and generative AI applications. It also introduces fine-tuning, parameter-efficient tuning, prompt engineering, retrieval-augmented generation (RAG), multimodal systems and evaluation. Examples use concepts applicable to both TensorFlow and PyTorch, although the implementation illustrations use PyTorch-style notation.

Learning Objectives

By the end of this chapter, a student should be able to:

1. Explain the purpose and operation of scaled dot-product attention and multi-head attention.

2. Distinguish encoder-only, decoder-only and encoder–decoder transformer architectures.

3. Describe how pretrained foundation models are adapted to specific tasks.

4. Design prompts and retrieval-augmented generation pipelines.

5. Compare full fine-tuning with parameter-efficient methods such as adapters and LoRA.

6. Implement the conceptual stages of a text or multimodal application.

7. Select suitable evaluation methods and identify reliability, safety and governance risks.

Why Attention Was Needed

Earlier sequence models, including recurrent neural networks (RNNs) and long short-term memory networks (LSTMs), processed tokens sequentially. This created two important limitations. First, training was difficult to parallelise because the hidden state at time step \(t\) depended on the state at time step \(t-1\). Second, information from distant tokens could be weakened as it passed through many recurrent steps.

Attention provides a direct, content-dependent connection between positions. For example, in the sentence “The animal did not cross the road because it was injured,” a model may need to relate “it” to “animal.” Attention allows the representation of a token to use information from relevant tokens, even when they are far away.

Attention is not simply a fixed importance score assigned by a human. It is a learned operation whose weights depend on the current inputs and learned projection matrices.

Scaled Dot-Product Attention

Let an input sequence be represented by vectors. Attention uses three matrices:

• Queries (Q): what each position is seeking;
• Keys (K): attributes by which positions can be matched;
• Values (V): information returned after matching.

The standard scaled dot-product attention operation is

\[

\operatorname{Attention}(Q,K,V)=\operatorname{softmax}\left(\frac{QK^T}{\sqrt{d_k}}+M\right)V.

\]

Here, \(d_k\) is the dimensionality of each key vector, and \(M\) is an optional mask. The matrix \(QK^T\) contains pairwise compatibility scores. Division by \(\sqrt{d_k}\) prevents dot products from becoming excessively large as the key dimension increases. Softmax converts each row of scores into non-negative weights that sum to one. The mask may assign a very negative value to prohibited positions before softmax.

For a single query \(q\), keys \(k_j\), and values \(v_j\), the output is

\[

\sum_j \alpha_j v_j,

\qquad

\alpha_j=\frac{\exp(q\cdot k_j/\sqrt{d_k})}{\sum_l \exp(q\cdot k_l/\sqrt{d_k})}.

\]

The vectors must have compatible dimensions: if there are \(n_q\) queries and \(n_k\) keys, then \(QK^T\) has shape \(n_q\times n_k\). The attention output has one value-weighted vector for each query.

Self-Attention and Cross-Attention

In self-attention, Q, K and V are produced from the same sequence. Each token can therefore build a context-sensitive representation. In cross-attention, queries come from one sequence while keys and values come from another. In an encoder–decoder translation model, decoder queries can attend to encoder outputs representing the source sentence.

A causal mask prevents a decoder position from attending to future tokens. This is essential for autoregressive generation: when predicting token \(t\), the model may use tokens 1 through \(t-1\), but not the correct future tokens.

Multi-Head Attention

A single attention operation may focus on one type of relationship. Multi-head attention runs several attention operations in parallel. For head \(i\),

\[

\operatorname{head}_i=\operatorname{Attention}(QW_i^Q,KW_i^K,VW_i^V).

\]

The heads are concatenated and projected:

\[

\operatorname{MultiHead}(Q,K,V)=\operatorname{Concat}(\operatorname{head}_1,\ldots,\operatorname{head}_h)W^O.

\]

The matrices \(W_i^Q,W_i^K,W_i^V\) and \(W^O\) are learned parameters. Different heads may learn different patterns, such as syntactic dependencies, positional relations or semantic similarity. This interpretation should be treated cautiously: an attention map is not automatically a complete explanation of the model’s reasoning.

Worked Shape Example

Suppose a batch contains 32 sequences, each with 128 tokens, and the model dimension is 512. With 8 heads, each head has dimension \(512/8=64\). For one layer:

• Q, K and V can have shape \((32,8,128,64)\);
• the score matrix has shape \((32,8,128,128)\);
• after applying the scores to V, the head outputs have shape \((32,8,128,64)\);
• concatenation produces \((32,128,512)\).

This illustrates why attention can be computationally expensive for long sequences: the score matrix grows approximately with the square of sequence length.

Transformer Architecture

The original Transformer introduced a sequence-to-sequence design based primarily on attention rather than recurrence. A typical transformer block contains:

1. multi-head attention;

2. a position-wise feed-forward network;

3. residual connections;

4. layer normalisation;

5. dropout during training in many implementations.

A feed-forward sublayer is applied independently to each token position. A common form is

\[

\operatorname{FFN}(x)=\sigma(xW_1+b_1)W_2+b_2,

\]

where \(\sigma\) is a nonlinear activation such as GELU or ReLU. The same feed-forward parameters are used at every position, although each position receives a different contextual vector.

Residual connections add a sublayer input to its output, helping gradients flow through deep networks. Layer normalisation normalises features within an individual example and token representation, rather than computing statistics across the batch in the same manner as batch normalisation.

Because self-attention alone does not inherently encode token order, transformers require positional information. This may be supplied through fixed sinusoidal encodings, learned position embeddings or relative/rotary position methods. A model’s usable context length depends on its architecture, training and implementation; a longer context window does not guarantee equal-quality use of every token.

Three Major Transformer Families

Encoder-only models use bidirectional self-attention and produce contextual representations. They are well suited to classification, named-entity recognition and semantic similarity. A classification head may use a pooled representation or a designated special token.

Decoder-only models use masked self-attention and predict the next token. They are naturally suited to text generation, dialogue and code completion. During generation, a model repeatedly predicts a probability distribution for the next token and appends a selected token.

Encoder–decoder models first encode an input and then generate an output using masked decoder self-attention plus cross-attention to the encoder. Translation, summarisation and structured text transformation are common applications.

From Language Models to Foundation Models

A language model estimates a distribution over token sequences. In autoregressive form, the chain rule gives

\[

P(x_1,\ldots,x_T)=\prod_{t=1}^{T}P(x_t\mid x_1,\ldots,x_{t-1}).

\]

The model is trained by minimising negative log-likelihood, often called cross-entropy loss:

\[

\mathcal{L}=-\sum_{t=1}^{T}\log P(x_t\mid x_{<t}).

\]

The logarithm is commonly natural logarithm, and the loss is usually averaged over valid predicted tokens in implementation. The objective trains statistical prediction; it does not directly guarantee factuality, truth, fairness or understanding.

A foundation model is a broadly pretrained model that can be adapted to many downstream tasks. Pretraining may involve very large and heterogeneous datasets. Adaptation can occur through prompting, supervised fine-tuning, alignment procedures, retrieval or tool use. Scale can improve capabilities, but it also increases infrastructure cost, evaluation difficulty and the potential impact of errors.

Fine-Tuning Pretrained Models

Fine-tuning continues training from pretrained weights on a task-specific dataset. A supervised example may contain an input and a desired label or output. The workflow generally includes:

1. selecting a model whose architecture, licence and context capacity suit the task;

2. defining a tokenizer and data format;

3. splitting data into training, validation and test sets without leakage;

4. preparing labels and masking irrelevant loss positions;

5. selecting learning rate, batch size, sequence length and stopping criteria;

6. evaluating against a baseline and monitoring harmful or unexpected behaviour.

For classification, a pretrained encoder can be paired with a linear prediction head. For instruction tuning, examples commonly contain an instruction, optional context and an ideal response. In causal language-model training, loss is often computed only on response tokens so that the model is not unnecessarily trained to reproduce the prompt. The exact masking policy must match the intended task.

Practical PyTorch Pattern

A conceptual training fragment is:

```python

outputs = model(input_ids=input_ids,

attention_mask=attention_mask,

labels=labels)

loss = outputs.loss

loss.backward()

optimizer.step()

optimizer.zero_grad()

```

In production, this fragment requires careful handling of padding, mixed precision, gradient accumulation, checkpointing, reproducibility and validation. TensorFlow implementations use analogous model, loss, gradient and optimiser operations. Library APIs change, so practitioners should verify current documentation rather than copying outdated code blindly.

Parameter-Efficient Fine-Tuning

Full fine-tuning updates all model parameters and can require substantial memory. Parameter-efficient fine-tuning (PEFT) updates a small number of additional or selected parameters while freezing most pretrained weights.

Common approaches include:

• Adapters: small trainable modules inserted into transformer layers;
• Prefix or prompt tuning: trainable vectors that influence attention without changing the main model;
• LoRA: low-rank trainable updates to selected weight matrices;
• Quantised LoRA: combines reduced-precision base weights with low-rank trainable updates.

For LoRA, a frozen weight matrix \(W\) is effectively adapted as

\[

W'=W+\Delta W,

\qquad \Delta W=BA,

\]

where \(A\) and \(B\) are trainable matrices whose inner rank is much smaller than the dimensions of \(W\). A scaling factor is often applied in implementations. LoRA reduces trainable parameter count, but it does not remove the need for high-quality data, careful evaluation or sufficient memory for activations and optimiser states.

PEFT is valuable when several organisations or tasks need separate adaptations of one shared base model. It also supports faster experimentation and storage of multiple task-specific modules. However, adapters may perform differently from full fine-tuning, and incompatible modules or licences can complicate deployment.

Prompt Engineering

A prompt specifies the task, context, constraints and desired output. Effective prompts are not magic formulas; they reduce ambiguity and provide information the model can use.

Useful practices include:

• state the task and audience explicitly;
• provide relevant context and delimit it clearly;
• specify output format, such as JSON fields or a table schema;
• include representative examples when appropriate;
• ask the model to identify uncertainty rather than inventing evidence;
• separate instructions from untrusted retrieved or user-supplied text;
• validate outputs with a parser or business rule.

Zero-shot prompting gives instructions without examples. Few-shot prompting provides examples within the context. Role descriptions can establish style or perspective, but they do not create new permissions or reliable domain expertise. Increasing prompt length can also reduce performance by diluting relevant information and increasing cost.

Decoding affects output. Greedy decoding selects the highest-probability next token at each step. Beam search maintains several candidate sequences and is often useful for constrained sequence generation, although it is not universally best. Temperature rescales logits before softmax; for temperature \(T>0\), probabilities are computed from \(\operatorname{softmax}(z/T)\). Lower values make distributions sharper, while higher values increase variability. Top-k and nucleus (top-p) sampling restrict candidate tokens. These controls change randomness, not factual knowledge.

Retrieval-Augmented Generation

A model’s parameters are not a dependable database of current organisational information. RAG connects generation to an external collection of documents.

A typical pipeline is:

1. collect and govern source documents;

2. clean, segment and attach metadata;

3. compute embeddings for chunks;

4. index embeddings in a vector or hybrid search system;

5. retrieve relevant chunks for a user query;

6. optionally rerank results;

7. place evidence into a controlled prompt;

8. generate an answer with citations or source references;

9. evaluate both retrieval and answer quality.

If vectors are normalised, cosine similarity between query vector \(q\) and document vector \(d\) is

\[

\operatorname{cos}(q,d)=q\cdot d.

\]

Without normalisation, cosine similarity is \((q\cdot d)/(\|q\|\|d\|)\). Dot-product, cosine and Euclidean search are not interchangeable unless the indexing and training assumptions support that choice.

RAG reduces some forms of outdated knowledge and can improve traceability, but it does not guarantee correctness. Poor chunking, missing documents, irrelevant retrieval, stale indexes, access-control failures and prompt injection can all produce unsafe answers. Retrieval should respect document permissions before content reaches the model.

Multimodal Generative AI

Multimodal systems process more than one data modality, such as text, images, audio or video. A common design encodes each modality into vectors, projects them into a shared or compatible representation space, and uses a transformer to combine them. In a vision-language model, an image encoder may produce visual tokens that a language model attends to while generating text.

Applications include visual question answering, document understanding, image captioning, accessibility tools, industrial inspection and educational tutoring. Multimodal evaluation must consider both modalities: an answer may be fluent but fail to identify a small visual defect, misread a chart or overlook text in a document. Image and audio inputs also introduce privacy, copyright and biometric risks.

Applied Case: University Policy Assistant

Background. An illustrative university wishes to answer questions about assessment rules, attendance, appeals and laboratory safety. Its policies change each academic year, and students need answers outside office hours.

Problem. A general language model may provide plausible but outdated answers. Directly fine-tuning on policy documents could memorise an old version and would make updates expensive.

Analysis. The institution identifies authoritative sources, version dates, audiences and access restrictions. It observes that answers must cite the relevant policy and distinguish general rules from programme-specific exceptions.

Approach. The team builds a RAG service. Policies are converted to text with headings and page metadata, divided into semantically coherent chunks, embedded and indexed. At query time, access checks occur before retrieval. A reranker selects the most relevant passages. The prompt instructs the model to answer only from supplied evidence, cite document title and version, state when evidence is insufficient, and avoid exposing private student information. A small PEFT adapter is trained on examples of institutional tone and citation formatting, while current facts remain in the retrieval layer. A structured output validator checks that citations are present.

Outcome. The system improves answer currency and provides auditable sources. Evaluation finds that retrieval recall is high for common policies but lower for questions involving exceptions, so the team adds metadata filters and escalation to human staff. The system is released as an assistant, not as an autonomous decision-maker.

Learning points. Separating durable language behaviour from changing factual knowledge is useful. Retrieval quality, access control and escalation design are as important as model selection. A fluent unsupported answer is a failure even if users find it persuasive.

Evaluation of Generative Systems

Evaluation should reflect the application rather than rely on one general score.

Intrinsic language measures include perplexity, which is derived from average negative log-likelihood. Lower perplexity indicates better prediction on the evaluation distribution, but it does not directly measure helpfulness, truthfulness or safety. Token-level scores can also favour common wording over appropriate reasoning.

Task metrics include accuracy and macro-F1 for classification, exact match for some question-answering tasks, and word error rate for speech recognition. BLEU, ROUGE and related overlap measures can be useful for comparison but may penalise valid paraphrases and should not be treated as complete quality measures.

Human and model-assisted evaluation may assess correctness, relevance, coherence, groundedness, citation quality, harmlessness and instruction following. Model-based judges require calibration against human ratings and can inherit biases.

For RAG, evaluate retrieval separately using measures such as recall at \(k\), then evaluate whether answers are supported by retrieved evidence. For a binary relevance setting, recall at \(k\) is the proportion of relevant items retrieved among all relevant items for the query. A system can have excellent retrieval but still generate an unsupported answer; conversely, a model may answer correctly without retrieving the required evidence by relying on memorisation.

Robust evaluation includes adversarial prompts, distribution shifts, multilingual cases, ambiguous questions, long-context tests, privacy tests and refusal tests. Evaluation data must be kept separate from training data to avoid contamination.

Limitations, Ethics and Governance

Transformers and generative systems have important limitations:

• Hallucination: fluent but unsupported or false content;
• bias: inherited from data, labels, prompts and deployment context;
• prompt injection: untrusted content attempts to override instructions;
• privacy leakage: memorisation or inappropriate retrieval of sensitive data;
• copyright and provenance concerns: training and generated content may involve protected material;
• security risks: generated phishing, malware assistance or data exfiltration;
• resource consumption: training and inference require energy, hardware and maintenance;
• automation bias: users may over-trust confident outputs;
• distribution shift: performance may degrade when language, users or policies change.

Responsible deployment requires data minimisation, documented provenance, access control, human oversight, incident reporting, red-team testing, model and prompt versioning, audit logs and a clear route for correction. Organisations should assess whether a use case is high impact, whether personal data is involved, and which legal and institutional requirements apply. A technical model card or system card should document intended use, limitations, evaluation conditions and known risks.

Contemporary Directions and Best Practices

Current development trends include longer-context processing, sparse and mixture-of-experts architectures, smaller specialised models, multimodal foundation models, tool-using agents and on-device inference. These developments do not eliminate fundamental requirements for data governance and evaluation. Longer context may increase cost and does not ensure that the model will attend to the correct passage. Agents introduce additional risks because generated actions can affect external systems.

A disciplined project should:

1. define the user task and unacceptable failures before selecting a model;

2. establish a non-generative baseline where appropriate;

3. use authoritative, permission-aware data;

4. begin with prompting or RAG before expensive fine-tuning when knowledge changes frequently;

5. use PEFT when task adaptation is needed but full updates are impractical;

6. validate structured outputs programmatically;

7. evaluate quality, fairness, security, latency and cost together;

8. monitor production drift and user feedback;

9. keep humans responsible for consequential decisions.

Chapter Summary

Attention computes content-dependent combinations of value vectors using query–key compatibility scores. Scaling, softmax and masking make the operation numerically practical and architecturally useful. Multi-head attention allows several learned relational patterns to be processed in parallel. Transformers combine attention with feed-forward networks, residual connections, normalisation and positional information.

Encoder-only, decoder-only and encoder–decoder models support different tasks. Foundation models are broadly pretrained systems adapted through prompting, fine-tuning, retrieval and tool integration. Full fine-tuning can be expensive, whereas adapters and LoRA update fewer parameters. Prompt engineering reduces ambiguity but cannot supply missing facts. RAG connects generation to external evidence, yet retrieval quality, permissions and prompt injection must be managed. Multimodal models extend these ideas across images, audio and other inputs.

Generative AI must be evaluated using task-appropriate automatic and human measures, with separate attention to retrieval, grounding, safety and robustness. Hallucination, bias, privacy, copyright, security, cost and automation bias require governance as well as technical controls. Effective professionals treat model deployment as a socio-technical system rather than merely an exercise in selecting a neural network.

Key Terms

Attention: A learned operation that weights information from different positions in an input.

Causal mask: A mask preventing a token from attending to future positions.

Cross-attention: Attention in which queries and keys/values come from different sequences.

Embedding: A numerical vector representation of an object such as a token, image or document.

Foundation model: A broadly pretrained model adaptable to multiple downstream tasks.

Hallucination: Unsupported, fabricated or incorrect generated content.

Layer normalisation: Normalisation applied across features within an individual representation.

LoRA: A parameter-efficient method using low-rank trainable updates to frozen weights.

Multi-head attention: Parallel attention operations whose outputs are concatenated and projected.

PEFT: Parameter-efficient fine-tuning that updates only a small part of a model or adds small modules.

Prompt engineering: Designing instructions, context and output constraints for a model.

RAG: Retrieval-augmented generation, in which retrieved external evidence supports generation.

Self-attention: Attention in which queries, keys and values originate from the same sequence.

Softmax: A function converting scores into non-negative values that sum to one.

Temperature: A decoding parameter controlling the sharpness of a probability distribution.

Token: A unit produced by a model’s tokenisation process.

Transformer: An architecture based primarily on attention and position-wise feed-forward operations.

Vector database: A system designed to index and search numerical vector representations.

Review Questions

Short Answer Questions

1. What roles do queries, keys and values play in scaled dot-product attention?

2. Why is the factor \(\sqrt{d_k}\) used in attention?

3. Distinguish encoder-only and decoder-only transformer models.

4. What problem does a causal mask solve?

5. Define retrieval-augmented generation.

Long Answer Questions

1. Explain the architecture of a transformer block, including attention, feed-forward layers, residual connections and normalisation.

2. Compare full fine-tuning, adapters and LoRA in terms of parameters, flexibility and deployment considerations.

3. Explain how a university could design and govern a policy-question-answering RAG system.

4. Discuss why perplexity and text-overlap metrics are insufficient for evaluating a generative AI assistant.

5. Analyse the principal ethical, security and privacy risks of deploying a multimodal foundation model.

Analytical / Application Questions

1. A decoder produces an answer containing unsupported claims despite retrieving relevant documents. Identify at least three possible causes and propose an evaluation or engineering intervention for each.

2. Design a transformer-based solution for classifying customer-support messages and routing urgent cases. Justify the model family, data split, metrics and human oversight.

3. An organisation has a frequently changing knowledge base and limited GPU memory. Compare prompting, RAG, full fine-tuning and LoRA, and recommend a staged implementation plan.

Multiple Choice Questions with Answers

Question 1

What is the primary purpose of scaling the query–key dot product by the square root of the key dimension?

A. To prevent attention scores from becoming excessively large

B. To ensure every token receives equal attention

C. To remove the need for positional information

D. To convert values into binary vectors

Correct Answer: A. To prevent attention scores from becoming excessively large

Question 2

Which transformer family is naturally designed for autoregressive next-token generation?

A. Encoder-only

B. Decoder-only

C. Convolution-only

D. Autoencoder-only

Correct Answer: B. Decoder-only

Question 3

What does a causal mask prevent in a decoder?

A. Attention to padding tokens in every situation

B. Attention to tokens from another modality

C. Attention to future token positions

D. Attention to the immediately preceding token

Correct Answer: C. Attention to future token positions

Question 4

In cross-attention, where do the queries and keys/values typically originate?

A. Both always originate from the same sequence

B. Queries originate from one sequence and keys/values from another

C. Queries are random and keys are fixed constants

D. Keys originate from labels and values from the optimiser

Correct Answer: B. Queries originate from one sequence and keys/values from another

Question 5

Which method represents a LoRA-style update?

A. W' = W + BA, where A and B are low-rank trainable matrices

B. W' = W / 0, which removes the original weights

C. W' = W with all values replaced by labels

D. W' = softmax(W) without any trainable parameters

Correct Answer: A. W' = W + BA, where A and B are low-rank trainable matrices

Question 6

What is a central purpose of retrieval-augmented generation?

A. To replace tokenisation with image compression

B. To connect generation to external evidence

C. To guarantee that every generated statement is true

D. To eliminate the need for access control

Correct Answer: B. To connect generation to external evidence

Question 7

If query and document embeddings are normalised, cosine similarity can be computed as:

A. The dot product of the two vectors

B. The sum of their token counts

C. The difference between their sequence lengths

D. The product of their batch sizes

Correct Answer: A. The dot product of the two vectors

Question 8

Which evaluation property is especially important for a RAG system?

A. Whether answers are supported by retrieved evidence

B. Whether every answer uses the maximum possible temperature

C. Whether the model ignores document permissions

D. Whether retrieval is performed only after generation

Correct Answer: A. Whether answers are supported by retrieved evidence

Question 9

What is the main effect of increasing temperature during sampling?

A. It generally makes the output distribution more variable

B. It increases the model’s factual knowledge

C. It permanently updates model parameters

D. It guarantees grammatical correctness

Correct Answer: A. It generally makes the output distribution more variable

Question 10

Which statement best describes an encoder-only transformer?

A. It uses bidirectional context and is commonly adapted for classification

B. It can attend only to future tokens

C. It cannot use self-attention

D. It is restricted to image generation

Correct Answer: A. It uses bidirectional context and is commonly adapted for classification