Transformers and Pretrained Language Models

Introduction

Transformers are the dominant architecture for modern natural language processing (NLP). They support machine translation, text classification, summarisation, question answering, information extraction, dialogue systems and code generation. Unlike recurrent neural networks, transformers process all tokens in a sequence in parallel and use attention to determine which tokens are relevant to one another.

The transformer architecture also changed how NLP systems are developed. Instead of training a separate model from the beginning for every task, practitioners commonly begin with a large pretrained language model and adapt it to a particular application. This approach is called transfer learning. Adaptation may involve full fine-tuning, parameter-efficient methods, or prompt design.

This chapter develops the conceptual and practical foundations needed to use transformer models responsibly. It explains self-attention, positional encoding, encoder and decoder components, the BERT and GPT families, transfer learning, fine-tuning, parameter-efficient adaptation and prompting.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain why attention is useful for language modelling.
2. Calculate the scaled dot-product attention for a small example.
3. Describe positional encoding and the major components of a transformer.
4. Distinguish encoder-only, decoder-only and encoder–decoder model families.
5. Explain how BERT and GPT are pretrained and applied.
6. Select between full fine-tuning, parameter-efficient adaptation and prompting.
7. Design clear prompts and evaluate adapted language models.
8. Identify limitations, risks and governance requirements in practical NLP systems.

1. From Sequential Models to Attention

A sentence is an ordered sequence of tokens. Earlier NLP systems often used recurrent neural networks (RNNs), long short-term memory networks (LSTMs) or gated recurrent units (GRUs). These models read tokens sequentially and maintain a hidden state. Although effective, sequential computation limits training parallelism and makes it difficult to preserve information across very long sequences.

Attention provides a direct mechanism for relating one token to other tokens. For example, in “The student submitted the assignment because she had finished it,” resolving “she” requires information about “student,” while interpreting “it” requires information about “assignment.” Attention allows a representation of one token to assign high importance to relevant tokens, even when they are distant in the sequence.

A transformer replaces recurrence with attention and position information. During training, the tokens of a sequence can generally be processed simultaneously. This improves hardware utilisation and permits large-scale pretraining, although the attention calculation itself becomes expensive for very long sequences.

2. Self-Attention

Self-attention is an operation in which every token examines the other tokens in the same sequence. Each input representation is transformed into three vectors:

• A query, representing what the token is looking for.
• A key, representing what information the token offers for matching.
• A value, representing the information to be aggregated.

For a sequence, these vectors are collected into matrices Q, K and V. The standard scaled dot-product attention is:

Here, Q is the query matrix, K is the key matrix, V is the value matrix, Kᵀ is the transpose of K, and dₖ is the dimensionality of each key vector. The softmax function converts each row of scores into non-negative weights that sum to 1. The division by √dₖ prevents dot products from becoming excessively large when the key dimension increases; otherwise, softmax gradients may become very small.

The formula assumes that Q, K and V have compatible dimensions. If there are n tokens, Q and K have shape n × dₖ, while V has shape n × dᵥ. Therefore QKᵀ has shape n × n, and the output has shape n × dᵥ. The n × n score matrix represents token-to-token relationships.

Worked Example: Single-Head Attention

Suppose a query is Q = [1, 0], two keys are K₁ = [1, 0] and K₂ = [0, 1], and the corresponding values are V₁ = [10, 0] and V₂ = [0, 20]. Let dₖ = 2.

The unscaled dot products are:

• Q · K₁ = 1
• Q · K₂ = 0

After scaling by √2, the scores are approximately 0.707 and 0. The softmax weights are approximately 0.67 and 0.33. Consequently, the output is:

0.67[10,0] + 0.33[0,20] ≈ [6.7,6.6]

The result is closer to V₁ because the query is more similar to K₁. This example is illustrative; real transformer layers use learned projections and many simultaneous queries.

Masking

A mask prevents attention from using particular positions. Padding masks stop a model from attending to artificial padding tokens. A causal, or look-ahead, mask prevents a decoder from seeing future tokens. In a causal sequence of length n, position i may attend only to positions j such that j ≤ i. This restriction is essential for autoregressive generation, because the model must predict the next token using only previously available information.

Multi-Head Attention

Instead of using one attention operation, transformers use several attention heads. Each head has separate learned projections:

where each head is an attention operation using its own projections, h is the number of heads, Concat joins the head outputs, and Wᴼ is an output projection matrix. Different heads may learn different relationships, such as syntactic dependencies, agreement, coreference or local phrase structure. Multi-head attention does not guarantee that every head has a simple human-interpretable role; interpretation requires analysis.

3. Positional Information

Self-attention alone is permutation-equivariant: if token representations are rearranged and no positional information is supplied, the operation cannot inherently distinguish “dog bites man” from “man bites dog.” Transformers therefore incorporate token position.

Sinusoidal Positional Encoding

The original transformer used fixed sinusoidal encodings. For position pos and dimension index i, one common definition is:

Here pos is the token position, i is a dimension index, and d is the model dimension. The encodings are added to token embeddings. The sine and cosine functions have different wavelengths, allowing the model to represent relative positional patterns. The formula assumes consistent indexing and that positions are measured as integer token locations; the encoding itself is dimensionless.

Modern models may instead use learned position embeddings, relative position representations or rotary positional embeddings. These alternatives can improve extrapolation or make relative distance easier to represent, but their behaviour depends on implementation and training conditions.

4. Transformer Architecture

A transformer layer usually contains attention, a position-wise feed-forward network, residual connections and layer normalisation. A simplified encoder layer can be represented as:

1. Compute self-attention.
2. Add the attention output to the input through a residual connection.
3. Apply layer normalisation.
4. Apply a feed-forward network independently to each token position.
5. Add another residual connection and normalise again.

A typical feed-forward network is:

where x is a token representation, W₁ and W₂ are learned weight matrices, b₁ and b₂ are bias vectors, and σ is a nonlinear activation such as GELU or ReLU. The same feed-forward parameters are applied at every sequence position, while each position has a different input representation.

Residual connections preserve information and improve gradient flow. Layer normalisation stabilises the distribution of activations within each token representation. Implementations differ in whether normalisation is placed before or after each sublayer, so descriptions of “the transformer layer” should specify the variant when implementation details matter.

Computational Cost

For a sequence of length n and model dimension d, the attention score matrix has n² entries. The attention portion therefore has approximately quadratic dependence on sequence length, commonly summarised as O(n²d) for the matrix multiplications. This is a computational and memory limitation for long documents. Sparse attention, linear-attention approximations, chunking, retrieval and long-context architectures are responses to this limitation, but each introduces trade-offs.

5. Model Families

Encoder-Only Models: BERT

BERT, or Bidirectional Encoder Representations from Transformers, uses transformer encoder layers. During pretraining, the model commonly uses masked language modelling: some input tokens are hidden or replaced, and the model predicts them using both left and right context. This bidirectional representation is useful for classification, named-entity recognition, semantic similarity and extractive question answering.

A classification task can use a special classification representation, often associated with a beginning-of-sequence token. A task-specific output layer is placed on top and trained using labelled examples. BERT-style models are generally less natural for unconstrained text generation because their pretraining objective is not left-to-right generation.

Decoder-Only Models: GPT

GPT, or Generative Pretrained Transformer, uses decoder-style layers with causal masking. Its autoregressive objective predicts the next token:

Here xᵢ is the token at position i. The product rule expresses the probability of a sequence as the product of conditional next-token probabilities. During training, the model usually minimises the negative log-likelihood:

The logarithm is natural logarithm unless otherwise specified. The loss is computed over target tokens, often averaged over non-padding positions in practical implementations. GPT-style models are naturally suited to continuation, dialogue, summarisation and code generation.

Encoder–Decoder Models

Encoder–decoder models contain an encoder that reads the input and a decoder that generates an output. The decoder uses masked self-attention over previously generated tokens and cross-attention to encoder representations. This design is effective for sequence-to-sequence tasks such as translation and abstractive summarisation.

These categories describe architectural tendencies, not absolute limitations. Model capabilities also depend on data, scale, objective, instruction training, tokenisation and evaluation.

6. Tokenisation and Representations

Language models operate on tokens rather than words in the ordinary linguistic sense. A tokenizer may use whole words, subwords, characters or byte-level units. Subword methods reduce the problem of unknown words and represent rare terms by combining smaller units. However, tokenisation affects sequence length, computational cost, spelling behaviour and performance in languages with different writing systems.

Each token is mapped to a learned embedding vector. The transformer then changes these embeddings into contextual representations. Thus, the representation of “bank” can differ in “river bank” and “bank account.” Embeddings are not definitions or guarantees of meaning; they are task-dependent numerical representations learned from data.

7. Pretraining and Transfer Learning

Pretraining exposes a model to a large corpus using a self-supervised objective, meaning that training targets are derived from the input rather than manually assigned labels. The pretrained model learns statistical regularities, including lexical associations, syntax and some domain patterns. It does not automatically acquire reliable factual knowledge, human values or genuine understanding.

Transfer learning adapts this general model to a downstream task. A typical workflow is:

1. Define the task, users, risks and success criteria.
2. Select a model family and tokenizer.
3. Prepare representative, legally usable data.
4. Establish a simple baseline.
5. Adapt the model using prompting, a task head, fine-tuning or an adapter.
6. Evaluate on held-out and stress-test data.
7. Deploy with monitoring, access controls and an escalation process.

Full Fine-Tuning

In full fine-tuning, all or most pretrained parameters are updated using task-specific examples. It can provide strong performance when sufficient high-quality data and computational resources are available. Risks include overfitting, catastrophic forgetting, training-data leakage and expensive storage of separate model copies.

A training objective for supervised classification may be cross-entropy:

where c indexes classes, y_c is 1 for the correct class and 0 otherwise in one-hot encoding, and p_c is the model’s predicted probability. The probabilities satisfy p_c ≥ 0 and ∑c p_c = 1. For multiple training examples, the loss is normally averaged across examples.

Parameter-Efficient Adaptation

Parameter-efficient fine-tuning updates only a small number of additional or selected parameters. Common approaches include:

• Adapters: small trainable modules inserted into frozen transformer layers.
• LoRA: low-rank updates to selected weight matrices.
• Prefix or prompt tuning: trainable vectors that influence attention without changing the main model.
• Quantised adaptation: reduced-precision base weights combined with trainable adapters.

In a simplified LoRA formulation, a weight update is represented as:

where W is the frozen original matrix, B and A are trainable matrices, and the intermediate rank is much smaller than the dimensions of W. If W has dimensions m × n and the rank is r, the additional parameters are r(m + n), rather than mn, ignoring optional scaling and bias terms. The low-rank assumption is a design choice, not a universal property of all useful updates.

Parameter-efficient methods reduce storage and computational requirements and make it easier to maintain different task or customer adapters. They do not remove the need for careful data governance or evaluation.

8. Prompt Design

A prompt is an instruction and context supplied to a language model. Effective prompt design is an engineering practice rather than a guarantee of correctness. A useful prompt normally specifies:

• The task and intended audience.
• Relevant context and permitted sources.
• Required output format.
• Constraints, such as length or language.
• Examples when the task is difficult or formatting-sensitive.
• A policy for uncertainty, such as “state that information is unavailable when evidence is insufficient.”

A structured prompt might contain a role or task description, input delimiters, output schema and evaluation criteria. Few-shot prompting supplies examples of inputs and desired outputs. Chain-of-thought-style internal reasoning should not be treated as a guarantee of validity; for professional systems, concise rationales, citations, intermediate checks or tool-verifiable outputs are often safer and easier to audit.

Prompt injection is an attack in which untrusted text attempts to override instructions. Systems should separate trusted instructions from retrieved or user-provided content, restrict tool permissions, validate outputs and avoid placing secrets in the model context.

9. Practical Application: Student Support Classifier

The following is an illustrative case, not a report of a real institution.

Background and Problem

A college receives thousands of student messages concerning enrolment, fees, examinations and technical support. Staff want to route messages to the correct department and identify urgent cases. A model must work with informal language, spelling errors and multilingual messages. Incorrect routing could delay assistance, so automation must include human review.

Analysis

The task is primarily classification, making an encoder-only model a reasonable starting point. The institution collects a representative sample, removes unnecessary personal information, defines categories and labels messages with agreement checks. A time-based test set is held out to detect performance changes after new academic policies are introduced.

Useful metrics include accuracy, macro-averaged F1 and per-class recall. Accuracy is:

Macro F1 gives each class equal importance. For a class, precision is the proportion of predicted positives that are correct, while recall is the proportion of actual positives that are found. The F1 score is:

where P is precision and R is recall; this formula applies when P + R is non-zero. Metrics should be calculated per class before macro averaging when class imbalance is important.

Approach and Solution

The team first creates a keyword and logistic-regression baseline. It then evaluates a pretrained encoder with a classification head. If labelled data are modest, parameter-efficient adaptation may offer a practical balance between performance and resource use. Messages containing threats, self-harm indications, legal complaints or uncertain predictions are routed to trained staff rather than automatically resolved.

The deployed system records model version, input category, confidence, decision and human outcome, while minimising retention of message content. Periodic audits examine performance by language, disability-related vocabulary and department. Staff can correct labels, and these corrections become candidates for future retraining after review.

Outcome and Learning Points

The central outcome is not simply a high aggregate score. A reliable service requires category-level evaluation, human escalation, privacy protection, monitoring for policy drift and a clear process for correcting errors. The case demonstrates that model choice, data quality, workflow design and governance are inseparable.

10. Evaluation and Deployment Best Practices

Evaluation should reflect actual use. Separate training, validation and test data to reduce optimistic estimates. Avoid leakage, such as placing nearly identical messages in both training and test sets. For generation, evaluate factuality, relevance, completeness, toxicity, refusal behaviour and formatting, using both automated measures and human review where appropriate.

Confidence scores are not automatically calibrated probabilities. A model may be highly confident and wrong. Retrieval-augmented generation can provide relevant external evidence, but retrieval quality, source authority and citation correctness must also be evaluated. Temperature and sampling settings affect output variability; lower randomness may improve consistency but does not guarantee truth.

Operational practices include version control for models and prompts, reproducible preprocessing, rate limits, access control, logging designed around privacy, rollback procedures and monitoring for distribution shift. Human oversight should be meaningful: reviewers need authority, time and sufficient information to challenge automated recommendations.

11. Limitations, Ethics and Governance

Transformer models can produce hallucinations, stereotypes, offensive content and confident errors. Training data may contain copyrighted material, personal data or social biases. Models can also memorise rare information. Organisations should establish a lawful basis for data use, follow applicable privacy and data-protection requirements, minimise personal data, document retention and provide appropriate user notice.

High-impact uses such as education, employment, healthcare, credit or public services require stronger safeguards. Impact assessments should consider affected groups, contestability, accessibility and human accountability. Security risks include prompt injection, data poisoning, model extraction and inadvertent disclosure through outputs or logs.

Environmental and financial costs also matter. Model size, training duration, inference frequency and hardware affect energy use and cost. Smaller models, caching, quantisation, batching and parameter-efficient adaptation can reduce resource consumption when they meet quality requirements.

12. Contemporary Developments

Current NLP systems increasingly combine language models with retrieval, external tools, structured output constraints and multimodal inputs. Long-context models attempt to process more tokens, but a larger context window does not ensure that every relevant fact is used correctly. Mixture-of-experts architectures activate only some parameters for each input, potentially improving efficiency while increasing systems complexity.

Instruction tuning trains models to follow natural-language tasks, while preference optimisation uses ranked or preference-labelled responses to shape behaviour. These methods can improve usability but may also introduce undesirable biases or make evaluation more difficult. Open-weight models support local deployment and inspection, whereas hosted models may provide convenient scale and maintenance. The appropriate choice depends on privacy, cost, latency, control, licence terms and risk.

Chapter Summary

Transformers use self-attention to relate tokens directly and process sequences in parallel. Scaled dot-product attention combines query–key similarity with value aggregation, while masks control which positions may interact. Positional information is necessary because attention alone does not represent order. Multi-head attention, residual connections, layer normalisation and feed-forward networks form the principal transformer components.

Encoder-only models such as BERT are strong for contextual representation and classification. Decoder-only models such as GPT use causal prediction and are naturally suited to generation. Encoder–decoder models support input-to-output transformations such as translation. Pretraining enables transfer learning, which may be implemented through full fine-tuning, parameter-efficient adaptation or prompt design. Successful professional deployment requires representative evaluation, privacy and security controls, human oversight, monitoring and explicit management of uncertainty.

Key Terms

Self-attention: Attention in which queries, keys and values are derived from the same sequence.

Query: A vector representing the information a token seeks.

Key: A vector used to match a token against a query.

Value: The information weighted and aggregated by attention.

Softmax: A function that converts scores into non-negative weights summing to one.

Causal mask: A mask preventing a position from attending to future positions.

Multi-head attention: Several attention operations whose outputs are concatenated.

Positional encoding: Information representing token order or relative position.

Residual connection: A shortcut that adds a layer input to its output.

Layer normalisation: Normalisation applied within a token representation.

Tokenizer: A procedure that converts text into model tokens.

BERT: An encoder-only transformer pretrained primarily with masked language modelling.

GPT: A decoder-only transformer family trained for autoregressive language prediction.

Transfer learning: Adapting knowledge from pretraining to a downstream task.

Fine-tuning: Updating model parameters using task-specific data.

Parameter-efficient fine-tuning: Adaptation that updates only a small parameter set.

LoRA: A low-rank method for representing trainable weight updates.

Prompt: Instructions and context supplied to a language model.

Prompt injection: An attempt by untrusted input to manipulate system instructions.

Hallucination: A plausible but unsupported or incorrect model output.

Review Questions

Short Answer Questions

1. What problem does positional encoding solve in a transformer?
2. Distinguish a query, key and value in self-attention.
3. Why is a causal mask used in a decoder-only language model?
4. State one typical application of BERT and one typical application of GPT.
5. What is the main purpose of parameter-efficient fine-tuning?

Long Answer Questions

1. Explain scaled dot-product attention, including the purpose of the scaling factor and softmax.
2. Compare encoder-only, decoder-only and encoder–decoder transformer architectures.
3. Describe the stages of a responsible transfer-learning workflow for an NLP application.
4. Explain full fine-tuning, adapters, LoRA and prompt tuning, including their practical trade-offs.
5. Discuss the main ethical, privacy and security risks of deploying language models.

Analytical / Application Questions

1. A college wants to classify student messages into six departments but has limited labelled data and strict privacy requirements. Recommend a model family and adaptation strategy, and justify your choice.
2. Design a prompt for extracting a complaint category, urgency level and supporting quotation from a customer message. Specify how the system should respond when evidence is insufficient.
3. A generation system has high overall accuracy but poor recall for messages written in a minority language. Propose an evaluation and improvement plan that addresses data, metrics, human review and governance.

Multiple Choice Questions with Answers

Question 1

What is the primary purpose of the scaling factor √dₖ in scaled dot-product attention?

A. To increase the sequence length

B. To prevent attention scores from becoming excessively large

C. To remove the need for positional encoding

D. To convert values into one-hot vectors

Correct Answer: B. To prevent attention scores from becoming excessively large

Question 2

Which model family is most naturally designed for left-to-right text generation?

A. Encoder-only

B. Decoder-only

C. A feed-forward classifier

D. A tokenizer

Correct Answer: B. Decoder-only

Question 3

What does a causal mask prevent during autoregressive generation?

A. Attention to padding tokens only

B. Use of the token embedding

C. Attention to future tokens

D. Use of multiple attention heads

Correct Answer: C. Attention to future tokens

Question 4

Which pretraining objective is strongly associated with BERT?

A. Masked language modelling

B. Image reconstruction

C. Sorting words alphabetically

D. Next-frame prediction

Correct Answer: A. Masked language modelling

Question 5

In the expression W′ = W + BA used for LoRA, what is the main advantage of B and A being low rank?

A. They require fewer trainable parameters

B. They eliminate all model bias

C. They guarantee factual outputs

D. They make tokenisation unnecessary

Correct Answer: A. They require fewer trainable parameters

Question 6

If a classifier makes 90 correct predictions out of 100, its accuracy is:

A. 0.09

B. 0.50

C. 0.90

D. 9.00

Correct Answer: C. 0.90

Question 7

Which architecture is generally best suited to translation between an input sequence and an output sequence?

A. Encoder–decoder

B. Decoder-only without attention

C. Tokenizer-only

D. Encoder-only with no output layer

Correct Answer: A. Encoder–decoder

Question 8

What is prompt injection?

A. Compressing a model into fewer parameters

B. An attempt by untrusted input to override or manipulate instructions

C. Adding positional encodings to embeddings

D. Converting words into subword tokens

Correct Answer: B. An attempt by untrusted input to override or manipulate instructions

Question 9

Which metric is especially useful for giving each class equal importance in an imbalanced classification problem?

A. Macro-averaged F1

B. Sequence length

C. Training batch size

D. Embedding dimension

Correct Answer: A. Macro-averaged F1

Question 10

Why is human escalation important in a high-impact NLP system?

A. It guarantees that the model never makes an error

B. It permits review of uncertain or potentially harmful decisions

C. It removes the need for testing

D. It makes all prompts identical

Correct Answer: B. It permits review of uncertain or potentially harmful decisions