Transformers, Pretrained Models and Large Language Models

Introduction

Natural Language Processing (NLP) has moved from systems built around manually designed features and task-specific models toward general-purpose neural models pretrained on very large text collections. The transformer is central to this transition. It provides an efficient mechanism for modelling relationships among tokens, supports parallel training, and serves as the foundation for models such as BERT, T5, GPT, and contemporary multimodal language systems.

A transformer is not synonymous with a large language model (LLM). A transformer is an architecture; pretraining is a learning strategy; and an LLM is usually a large pretrained generative language model. Understanding these distinctions is essential for selecting, adapting, evaluating, and governing NLP systems.

Learning Objectives

By the end of this chapter, a reader should be able to:

• explain transformer encoder, decoder, and encoder–decoder architectures;
• derive and interpret scaled dot-product self-attention;
• explain positional representations and their limitations;
• distinguish autoregressive, masked-language, and sequence-to-sequence pretraining;
• analyse scaling, transfer learning, prompting, and instruction tuning;
• compare full fine-tuning with parameter-efficient adaptation;
• select models using task, data, latency, cost, safety, and governance criteria;
• identify reliability, bias, privacy, and security risks in LLM deployment.

1. From Sequential Networks to Transformers

Recurrent neural networks process tokens sequentially. Although gated variants such as LSTMs improve long-range memory, sequential computation limits training parallelism and makes very distant interactions difficult to learn. Convolutional models improve parallelism but require multiple layers or large kernels to connect distant positions.

Transformers replace recurrence with attention. Every token can directly exchange information with every other token in a layer. During training, all positions in a sequence can generally be processed in parallel. The principal trade-off is that standard full attention has quadratic time and memory growth with sequence length: for a sequence of length n, the attention score matrix contains n² entries.

A transformer commonly contains repeated layers. Each layer includes attention, a position-wise feed-forward network, residual connections, and normalization. The architecture is modular: an encoder represents input sequences, a decoder generates output sequences, and an encoder–decoder model maps one sequence to another.

2. Self-Attention

Let the input representation be a matrix X ∈ ℝⁿˣᵈ, where n is the number of tokens and d is the representation width. A learned linear transformation produces queries, keys, and values:

Here W_Q, W_K, and W_V are learned parameter matrices. Queries express what a position is seeking, keys describe what each position offers for matching, and values contain information to be aggregated. If the query and key dimensions are dₖ, scaled dot-product attention is:

The product QKᵀ produces pairwise compatibility scores. Division by √dₖ prevents dot products from becoming excessively large when dₖ increases; without scaling, the softmax may become very peaked and gradients may become less useful. Softmax is applied row by row, so each query position obtains a distribution of non-negative weights summing to 1. The output is a weighted combination of value vectors.

The formula assumes compatible matrix dimensions and applies to dense attention over a sequence. A causal mask modifies the scores so that a token cannot attend to future tokens. Padding masks prevent attention to artificial padding positions. A masked score is commonly assigned a very negative value before softmax.

Multi-Head Attention

Instead of using one attention operation, multi-head attention uses h heads. Each head has its own projections and can learn different relations, such as syntactic agreement, coreference, or positional proximity:

The heads are concatenated and projected:

If the model width is d_model and heads divide that width evenly, a common design uses dₖ = d_model ÷ h. This is a design convention, not a universal requirement.

Transformer Layer

A simplified encoder layer can be expressed as:

The exact placement of normalization varies between architectures. The feed-forward network is applied independently at each position:

W₁ and W₂ are learned matrices, b₁ and b₂ are biases, and σ is a nonlinear activation such as GELU or a gated alternative. Residual connections help preserve information and improve optimization.

3. Positional Information

Self-attention alone is permutation-equivariant: if token order is changed and no positional signal is supplied, the model cannot inherently distinguish the resulting order. Positional information is therefore essential for language.

Absolute Positional Representations

A simple method adds a learned position embedding p_i to token embedding e_i:

The model learns a vector for each supported position. Sinusoidal encodings instead use fixed functions with position pos and dimension index i:

These functions provide varied frequencies and can represent relative offsets through trigonometric relationships. Learned and sinusoidal schemes may still face difficulties when inference sequences exceed training lengths.

Relative and Rotary Methods

Relative-position methods represent distance or relative displacement rather than only absolute index. Rotary positional embeddings rotate query and key components by position-dependent angles, causing their dot products to encode relative position. Such methods are widely used in modern decoder models, but long-context extrapolation remains an engineering and research challenge.

4. Transformer Architectures

BERT is representative of encoder-only modelling. GPT-style systems are decoder-only. T5 treats many tasks as text-to-text and uses an encoder–decoder design. Architecture should be selected according to the direction of information flow required by the task, not merely model popularity.

5. Tokenisation and Representations

Language models operate on tokens rather than abstract words. Subword methods, including byte-pair encoding and unigram approaches, balance vocabulary size with the ability to represent rare words. Tokenisation affects cost, context length, spelling robustness, and multilingual performance. A word may be represented by one token in one language and many tokens in another.

A token index is mapped to an embedding vector. Through successive layers, the representation becomes contextual: the vector for a word depends on surrounding tokens. This differs from static embeddings, where one word has one principal vector regardless of context.

6. Pretraining Objectives

Autoregressive Language Modelling

A decoder language model estimates a sequence probability using the chain rule:

The model is trained by minimizing negative log-likelihood, often equivalent to cross-entropy over the next-token distribution:

The logarithm is usually natural, and x₍<ᵢ₎ denotes preceding tokens. This objective supports generation but does not guarantee factuality or truthfulness.

Masked Language Modelling

An encoder model masks selected input tokens and predicts them from both left and right context. It is effective for contextual representation learning, but its pretraining task differs from free-form generation.

Denoising and Sequence-to-Sequence Learning

An input is corrupted by masking, deleting, or permuting spans, and the model reconstructs the original text. This trains an encoder–decoder system to transform corrupted sequences into clean sequences and is useful for translation, summarisation, and instruction-style tasks.

Data Preparation

Pretraining quality depends on data quality, not only quantity. Deduplication, language identification, removal of malware or personal data where appropriate, document quality filtering, licence analysis, and contamination checks are important. Training data can encode social stereotypes, copyrighted material, private information, and low-quality synthetic text.

7. Scaling and Emergent Capability

Increasing parameters, training tokens, and computation often improves loss and downstream performance, but improvements are not automatic. Training must balance model size and data; a very large model trained on insufficient or poor-quality data may be inefficient. Compute-optimal training treats parameter count and token count as coupled decisions rather than assuming that larger is always better.

A useful systems approximation is:

This is an engineering estimate, not a universal physical law. Actual cost depends on sequence length, hardware utilization, precision, communication, optimizer state, and implementation. Inference cost additionally depends on generated length and, for autoregressive models, the key–value cache.

Scaling curves often show approximately smooth improvement in loss, while capabilities such as tool use or multi-step reasoning may appear discontinuous under particular evaluations. Such apparent emergence can reflect thresholds, prompting, measurement choice, or benchmark effects; it should not be treated as proof of human-like understanding.

8. Adaptation: Fine-Tuning, Prompting, and Instruction Tuning

Supervised Fine-Tuning

In supervised fine-tuning, pretrained parameters are updated using labelled examples. The dataset should reflect the intended task, input distribution, output format, and error costs. A validation set supports hyperparameter selection, while a held-out test set should be used only for final evaluation.

Full fine-tuning can produce strong task performance but requires storing and serving a separate copy of many parameters for each task. It can also cause catastrophic forgetting, overfitting, or unsafe behavioural changes.

Prompting and In-Context Learning

Prompting specifies a task through instructions, demonstrations, constraints, and output schemas. Zero-shot prompting gives no examples; few-shot prompting supplies examples in context. Prompt quality involves clear delimiters, explicit criteria, representative examples, and refusal or uncertainty requirements where appropriate.

In-context learning changes the model’s behaviour without updating its weights. It is convenient, but performance can be sensitive to example order, wording, token budget, and distribution shift.

Instruction Tuning and Preference Optimisation

Instruction tuning uses examples of instructions and desirable responses to make a pretrained model more useful for human-directed tasks. Preference-based methods train a reward or preference model, or directly optimise comparisons between responses. Human feedback can improve helpfulness, but annotator values, majority preferences, and reward-model errors can introduce new biases. Alignment should therefore be evaluated rather than assumed.

Parameter-Efficient Adaptation

Parameter-efficient methods update a small fraction of parameters. In low-rank adaptation (LoRA), an update to a weight matrix is represented approximately as:

where A and B have a chosen low rank r, typically much smaller than the original matrix dimensions. The base model remains frozen while A and B are trained. Adapters, prefix tuning, prompt tuning, and quantised low-rank methods reduce storage and memory requirements, although they do not eliminate data, evaluation, or inference costs.

9. Generation and Decoding

At each step a model produces a probability distribution over the vocabulary. Greedy decoding selects the highest-probability token. Beam search maintains several partial sequences and is useful for some sequence-to-sequence tasks, though it can produce generic outputs. Sampling introduces controlled variation.

Temperature rescales logits before softmax:

Here z_i is the logit for token i and T > 0 is temperature. Lower T makes the distribution sharper; higher T makes it flatter. Top-k sampling restricts choices to k tokens, while nucleus or top-p sampling chooses the smallest candidate set whose cumulative probability reaches p. These controls affect diversity, not factual correctness.

10. Retrieval-Augmented Generation and Tool Use

A language model’s parameters are not a reliable database. Retrieval-augmented generation (RAG) retrieves relevant documents, places them in the model context, and asks the model to answer using that evidence. A practical pipeline includes document ingestion, chunking, embedding, indexing, retrieval, reranking, prompt construction, generation, and citation verification.

RAG can improve freshness and domain grounding, but retrieval errors, irrelevant context, prompt injection in documents, and unsupported synthesis remain risks. Tool-using systems extend this pattern by allowing a model to call search, databases, calculators, or business APIs. Tools require authentication, input validation, permission boundaries, logging, and human approval for consequential actions.

11. Model Selection

Model selection is a constrained decision rather than a leaderboard exercise. Relevant criteria include:

Benchmark scores should be supplemented with task-specific tests, adversarial tests, calibration checks, latency measurements, and total-cost analysis. A smaller model with retrieval and domain adaptation may outperform a larger general model on a narrow enterprise task.

12. Evaluation and Reliability

Intrinsic loss does not fully measure practical quality. Evaluation may include exact match, precision, recall, F1, BLEU or ROUGE where appropriate, ranking metrics, factuality checks, human preference, calibration, and task-specific utility. For a binary classifier:

TP, FP, and FN denote true positives, false positives, and false negatives. These metrics assume a defined positive class and must be interpreted with prevalence and error costs.

Generative evaluation should test hallucination, refusal quality, instruction following, consistency, robustness to paraphrase, prompt injection, and sensitive-data leakage. Human evaluation requires clear rubrics, trained annotators, inter-rater analysis, and representative samples. Evaluation data must not silently overlap with training data, or scores may be inflated.

13. Ethical, Legal, and Governance Issues

LLMs can reproduce biased associations, generate persuasive falsehoods, expose memorised personal information, and enable fraud or disinformation. Developers should conduct data-provenance review, privacy impact assessment, risk classification, red-team testing, access control, incident response, and continuous monitoring.

Human oversight is especially important in employment, education, healthcare, credit, law enforcement, and other high-impact settings. Users should know when automated generation is involved, and organisations should preserve audit records sufficient to reconstruct important decisions. Governance should address model cards, dataset documentation, acceptable-use policies, vendor obligations, copyright and licensing, retention, security, and change management. Regulation differs by jurisdiction; legal review is therefore necessary before deployment.

14. Illustrative Case: Internal Policy Assistant

Background

A financial-services organisation wants an assistant that answers employee questions about internal policies. The source material changes monthly and contains confidential operational information.

Problem

A general LLM sometimes invents policy details, uses outdated information, and exposes documents to unauthorised users. Fully fine-tuning the model on policy text would not reliably solve freshness or access control.

Analysis

The principal requirements are grounded answers, document-level permissions, citations, rapid updates, low latency, and auditability. The organisation defines a test set of realistic questions, ambiguous questions, adversarial prompts, and questions for which the correct response is “insufficient information.”

Approach

The team deploys a moderately sized instruction-tuned model with RAG. Documents are versioned, access labels are attached to chunks, and retrieval is filtered by the user’s identity before generation. The prompt requires citations and prohibits unsupported claims. A cross-encoder reranker improves relevance. Low-rank adaptation is used only to teach the desired answer format, not to memorise policy content. Responses are evaluated for citation support, access violations, refusal quality, latency, and cost.

Outcome

The system is more maintainable than a periodically retrained model because policy updates enter the index after approval. It still routes high-risk questions to a human and refuses when evidence is missing. Monitoring detects retrieval failures and changes in question distribution.

Learning Points

The case demonstrates that model size is not the whole solution. Retrieval, permissions, evaluation, source governance, and escalation procedures are part of the NLP system. Adaptation should address behaviour, while authoritative and changing knowledge should remain in controlled external sources.

15. Best Practices

1. Define the task, users, risks, and success metrics before selecting a model.
2. Establish data provenance, licensing, privacy, quality, and contamination controls.
3. Use the smallest model that meets capability and reliability requirements.
4. Separate knowledge grounding from behavioural adaptation where possible.
5. Start with prompting and retrieval; use fine-tuning when evidence shows it is needed.
6. Prefer parameter-efficient adaptation when many task variants must be maintained.
7. Evaluate on representative, adversarial, multilingual, and out-of-distribution cases.
8. Measure factuality and harmful failure modes, not only average accuracy.
9. Apply least privilege to tools, data, and model actions.
10. Monitor production behaviour, document changes, and maintain rollback procedures.

Chapter Summary

Transformers use self-attention to connect tokens directly and process sequences in parallel. Queries, keys, and values produce attention weights, while multi-head attention captures several relation types. Positional representations supply order information. Encoder-only, decoder-only, and encoder–decoder architectures support different tasks.

Pretraining objectives include autoregressive prediction, masked-language modelling, and denoising sequence-to-sequence learning. Scaling parameters, data, and compute can improve capability, but data quality, evaluation, and systems constraints remain decisive. Prompting, supervised fine-tuning, instruction tuning, preference optimisation, and parameter-efficient methods provide distinct routes to adaptation. RAG and tools can ground responses in current information, but require security and governance controls. Effective model selection balances capability, cost, latency, privacy, safety, licensing, and operational reliability.

Key Terms

Transformer: Neural architecture based primarily on attention and feed-forward layers.

Self-attention: Attention in which queries, keys, and values come from the same sequence.

Query: Vector representing information sought by a position.

Key: Vector used to match a position with queries.

Value: Vector aggregated according to attention weights.

Multi-head attention: Parallel attention operations with separate projections.

Causal mask: Constraint preventing access to future tokens.

Positional encoding: Signal representing token order or relative position.

Tokenisation: Conversion of text into model-processing units.

Autoregressive modelling: Prediction of each token from preceding tokens.

Masked language modelling: Prediction of hidden tokens using surrounding context.

Instruction tuning: Supervised adaptation using instruction–response examples.

Prompting: Specifying a task through input text without changing model weights.

In-context learning: Behavioural adaptation from examples included in the prompt.

LoRA: Low-rank parameter-efficient adaptation method.

Quantisation: Representation of values with lower numerical precision.

Retrieval-augmented generation: Generation conditioned on retrieved external evidence.

Hallucination: Unsupported or fabricated model output.

Prompt injection: Input attempting to override intended instructions or controls.

Model card: Documentation describing a model’s properties, uses, and limitations.

Review Questions

Short Answer Questions

1. Why is the scaling factor √dₖ used in scaled dot-product attention?
2. Distinguish encoder-only and decoder-only transformer architectures.
3. What problem does positional encoding solve?
4. Define parameter-efficient fine-tuning and give one example.
5. Why can retrieval improve an LLM without changing its parameters?

Long Answer Questions

1. Explain the complete flow of multi-head self-attention and its integration with residual and feed-forward layers.
2. Compare autoregressive, masked-language, and denoising pretraining objectives, including suitable applications.
3. Analyse the benefits and limitations of prompting, supervised fine-tuning, instruction tuning, and LoRA.
4. Develop a model-selection framework for an enterprise NLP application.
5. Discuss reliability, privacy, bias, security, and governance risks in deploying an LLM.

Analytical / Application Questions

1. A team needs a current policy assistant with access-controlled documents. Should it fine-tune on all policies, use RAG, or combine methods? Justify the architecture and evaluation plan.
2. A model has high average accuracy but frequently fabricates answers on rare cases. Design an evaluation and mitigation programme.
3. Compare two candidate models: one large hosted model with high capability and one smaller locally deployed model with lower cost and stronger privacy. Specify the evidence needed to choose between them.

Multiple Choice Questions with Answers

Question 1

Which expression defines scaled dot-product attention?

A. softmax(QKᵀ ÷ √dₖ)V

B. softmax(Q + K + V)

C. QKᵀV without normalization

D. softmax(QVᵀ)K

Correct Answer: A. softmax(QKᵀ ÷ √dₖ)V

Question 2

What is the main purpose of a causal mask in a decoder?

A. To remove padding tokens from the vocabulary

B. To prevent a position from attending to future tokens

C. To force all attention weights to be equal

D. To increase the embedding dimension

Correct Answer: B. To prevent a position from attending to future tokens

Question 3

Which architecture is most directly associated with bidirectional masked-token representation learning?

A. Decoder-only

B. Encoder-only

C. Convolution-only

D. Retrieval-only

Correct Answer: B. Encoder-only

Question 4

In autoregressive language modelling, the model generally predicts:

A. All tokens without using context

B. The next token from preceding tokens

C. Only sentence length

D. A document’s author

Correct Answer: B. The next token from preceding tokens

Question 5

If a model has TP = 80 and FP = 20, its precision is:

A. 0.20

B. 0. cinquante

C. 0.80

D. 0.90

Correct Answer: C. 0.80

Question 6

Which method updates a low-rank pair of matrices while commonly keeping the base model frozen?

A. LoRA

B. Beam search

C. Tokenisation

D. Causal masking

Correct Answer: A. LoRA

Question 7

What is the principal role of retrieval-augmented generation?

A. To replace tokenisation with images

B. To ground generation in retrieved external information

C. To guarantee that every output is true

D. To eliminate the need for access control

Correct Answer: B. To ground generation in retrieved external information

Question 8

Increasing temperature during sampling generally makes the output distribution:

A. Sharper and more deterministic

B. Flatter and potentially more diverse

C. Uniformly zero

D. Independent of logits

Correct Answer: B. Flatter and potentially more diverse

Question 9

Which consideration is least appropriate as the sole basis for model selection?

A. Task-specific evaluation

B. Privacy and deployment constraints

C. A single unrelated leaderboard score

D. Latency and total cost

Correct Answer: C. A single unrelated leaderboard score

Question 10

Why is data provenance important for LLM development?

A. It determines the screen brightness of servers

B. It supports quality, licensing, privacy, and accountability analysis

C. It removes the need for testing

D. It guarantees emergent reasoning

Correct Answer: B. It supports quality, licensing, privacy, and accountability analysis