Model Adaptation, Agents and Multimodal AI

Introduction and Context

Large language models (LLMs) are powerful general-purpose systems, but a pretrained model is rarely optimal for a particular organisation, domain, workflow or interaction style. A medical assistant may need controlled terminology and cautious uncertainty statements; a financial assistant may need current market data and audit trails; an enterprise support agent may need access to internal documents and ticketing systems. Model adaptation is the set of methods used to make a general model more useful, reliable and controllable for a defined purpose.

This chapter examines four closely related adaptation strategies: full fine-tuning, parameter-efficient adaptation, synthetic-data generation and preference optimisation. It then develops the architecture of tool-using agents and explains how language models can be integrated with images, speech, video and structured sensor data. The central engineering question is not simply, “How can the model produce fluent text?” It is, “Which combination of data, parameters, tools, modalities and controls produces dependable behaviour under realistic constraints?”

Learning Objectives

By the end of this chapter, you should be able to:

1. Compare supervised fine-tuning, parameter-efficient methods, synthetic data and preference optimisation.
2. Select an adaptation strategy using quality, cost, latency, privacy and maintenance criteria.
3. Design a tool-using agent with planning, execution, observation, memory and safety controls.
4. Explain how text, image, speech, video and structured data can be aligned and fused.
5. Evaluate adapted and multimodal systems using task, safety, robustness and operational metrics.
6. Identify ethical, legal and governance risks in advanced generative-AI deployments.

1. The Adaptation Problem

Pretraining learns broad statistical regularities from large collections of text, code, images, audio or multimodal data. The resulting model is general but not necessarily aligned with a specific task. Adaptation changes either the model’s parameters, its inputs and prompts, its retrieval context, or its surrounding software.

A useful distinction is:

• Prompting: changing the input instructions without changing model parameters.
• Retrieval augmentation: supplying relevant external information at inference time.
• Adaptation: changing parameters or training behaviour so that desired responses become more likely.
• Orchestration: connecting the model to tools, memory, policies and other models.

Adaptation should be treated as a systems decision. Training a model may improve style and task consistency, but it does not automatically provide current facts. Retrieval may provide current facts, but it does not guarantee correct reasoning. A tool may calculate accurately, but the agent may invoke it incorrectly. Reliable systems therefore combine techniques rather than assuming that one method solves every failure mode.

2. Fine-Tuning and Parameter-Efficient Adaptation

2.1 Supervised fine-tuning

In supervised fine-tuning (SFT), a pretrained model is trained on examples of desired behaviour, usually pairs of instructions and responses. If the model defines a conditional probability pθ(y|x), where x is the input, y is the target output and θ represents model parameters, the common objective is next-token cross-entropy:

Here, D is the training set, T is the number of target tokens, y_t is token t, and y_<t denotes preceding target tokens. The loss is applicable when the target sequence is known and token-level likelihood is an appropriate training signal. In practice, padding tokens may be masked, and the loss may be averaged rather than summed.

SFT is effective for instruction following, structured output, domain terminology, classification, coding conventions and response formatting. Its limitations include high computational cost, catastrophic forgetting, sensitivity to noisy labels and the risk of learning undesirable shortcuts. Fine-tuning does not guarantee factuality: a model can become more confident without becoming more correct.

2.2 Full-parameter versus parameter-efficient training

Full fine-tuning updates most or all trainable parameters. It can produce substantial behavioural change but requires large memory and creates a separate model checkpoint for each task or customer.

Parameter-efficient fine-tuning (PEFT) updates only a small number of parameters while freezing the base model. Important methods include:

• Adapters: small trainable layers inserted into selected network blocks.
• LoRA: low-rank trainable updates applied to weight transformations.
• QLoRA: LoRA training while the base model is stored in a quantised representation.
• Prompt or prefix tuning: trainable vectors are added to the input or intermediate context.

For a base weight matrix W, LoRA commonly represents the update as:

where W is frozen, A and B are trainable low-rank matrices, and ΔW = BA. If W has dimensions d_out × d_in and the rank is r, then the additional parameter count is r(d_in + d_out), rather than d_outd_in. The method is useful when r is much smaller than both dimensions. The exact memory and quality benefits depend on target layers, quantisation, sequence length, optimiser state and hardware.

PEFT reduces storage and training costs, supports multiple task-specific adapters and can simplify deployment. However, an adapter is not automatically safe to merge with another adapter; conflicting behaviours, distribution shifts and version-management problems may arise. PEFT is especially attractive when a common base model serves multiple departments or clients.

2.3 Choosing an adaptation strategy

A practical selection process begins with a baseline. Measure the unadapted model on representative tasks, classify failures, and then choose the least expensive intervention that addresses the dominant failure. If errors arise from missing documents, fine-tuning may be inappropriate; retrieval is usually more direct. If errors arise from inconsistent formatting, SFT or constrained decoding may be preferable.

3. Synthetic Data and Preference Optimisation

3.1 Synthetic data

Synthetic data is generated by a model, simulator, program or transformation process rather than collected directly from human interactions. It can expand rare cases, create instruction-response pairs, produce privacy-preserving variants and support safety testing.

A dependable synthetic-data pipeline includes:

1. Define the target capability and unacceptable behaviours.
2. Generate candidate examples using templates, models, simulators or combinations of these.
3. Filter for validity, duplication, toxicity, privacy leakage and task relevance.
4. Verify samples with human experts, deterministic programs or independent models.
5. Maintain provenance: generator, prompt, source, date, licence and review status.
6. Evaluate on a held-out human-authored set.

Synthetic data can amplify model errors. If a generator produces stereotypical, factually incorrect or stylistically narrow examples, training on them may cause model collapse or reduce diversity. Synthetic examples should therefore supplement, not silently replace, high-quality human and naturally occurring data.

3.2 Preference optimisation

Preference optimisation trains a model to favour one response over another according to human or expert judgements. In preference data, a prompt x is associated with a preferred response y⁺ and a less preferred response y⁻. Direct Preference Optimisation (DPO), for example, uses a reference model and increases the relative likelihood of the preferred answer. A simplified DPO objective is:

Here, π_θ is the trainable policy, π_ref is a fixed reference policy, σ is the logistic sigmoid, β controls the strength of preference optimisation, and E denotes an average over preference examples. The equation assumes paired preferences and a reference-policy formulation; implementations may differ in masking, normalisation and data construction.

Another family uses reinforcement learning from human feedback (RLHF): a reward model is trained from preferences, and the language model is optimised to obtain high predicted reward while remaining close to a reference policy. Preference methods can improve helpfulness, tone, refusal behaviour and instruction following. They may also produce verbosity, sycophancy or reward hacking if the preference signal rewards superficial features rather than genuine quality.

Preference labels are not universal truth. They reflect annotator instructions, cultural assumptions, expertise and the information visible to the evaluator. High-stakes systems should use domain experts, disagreement analysis, calibration and explicit safety criteria.

4. Evaluation of Adapted Models

Evaluation must separate capability from deployment quality. Useful measures include exact match, precision, recall, F1, perplexity, pass@k for code, groundedness, citation correctness, refusal appropriateness, latency, cost and user success.

For binary classification, precision and recall are:

TP denotes true positives, FP false positives and FN false negatives. These measures apply when outputs can be mapped to binary decisions. They should not be treated as complete measures of generative quality.

An adaptation evaluation should include:

• A training set, validation set and contamination-controlled test set.
• In-domain and out-of-domain examples.
• Adversarial prompts and prompt-injection attempts.
• Human assessment with a documented rubric.
• Regression tests for previously successful capabilities.
• Cost, latency, throughput and failure-recovery measurements.

5. Tool-Using Agents

An agent is a software system in which a model selects actions to pursue a goal within an environment. A tool-using language agent normally contains:

1. Goal and policy layer: interprets objectives and constraints.
2. Planner: decomposes a task into steps.
3. Tool registry: describes available functions, schemas and permissions.
4. Executor: validates and invokes tools.
5. Observation handler: converts tool outputs into model-readable evidence.
6. Memory: stores selected conversation, facts or prior actions.
7. Guardrails: enforce authentication, policy, budgets and human approval.
8. Audit layer: records prompts, tool calls, outputs, versions and decisions.

A basic agent loop is: receive goal → inspect context → plan → request a tool call → validate arguments → execute → observe result → revise plan → answer or request approval. The model should not directly execute arbitrary generated code or commands. A typed tool schema, least-privilege credentials, input validation and timeouts are essential.

Planning patterns

• ReAct: alternate reasoning-oriented internal steps with actions and observations.
• Plan-and-execute: create a plan first, then execute steps, revising when necessary.
• Router architecture: choose among specialist models or tools.
• Workflow graph: represent allowed states and transitions explicitly.
• Multi-agent architecture: assign subtasks to specialised agents, with an orchestrator.

Open-ended autonomy is not always desirable. A deterministic workflow is safer for regulated or repetitive processes. Agents are most useful where tasks involve ambiguity, unstructured information and a manageable set of well-defined actions.

Tool safety

Tool descriptions and retrieved documents are untrusted inputs. A malicious document can contain instructions intended to override the agent’s policy, a problem known as indirect prompt injection. Controls include separating data from instructions, validating tool arguments, restricting network access, using allowlists, requiring confirmation for irreversible actions, detecting sensitive data exfiltration and limiting action budgets.

A high-impact action should use graduated autonomy:

• Read-only actions may be automatic.
• Reversible updates may require lightweight confirmation.
• Irreversible financial, legal or operational actions should require explicit human approval.

6. Multimodal AI

Multimodal AI processes or generates more than one modality, such as text, images, speech, video, tables or sensor streams. Integration can occur at different levels:

• Early fusion: combine modality representations before substantial reasoning.
• Intermediate fusion: use cross-attention or shared latent representations.
• Late fusion: process modalities separately and combine decisions or generated evidence.
• Tool-mediated fusion: use specialist models, such as OCR or speech recognition, and pass their outputs to a language model.

Text-image systems commonly encode an image into a sequence of visual embeddings and align those embeddings with text representations. Speech systems may use automatic speech recognition (ASR) to produce text, a speech encoder for direct acoustic representations, or a speech-capable generative model. Text-to-speech (TTS) then converts a response into audio. Video adds a temporal dimension and requires sampling, tracking or temporal attention.

A practical enterprise architecture often uses specialist components:

1. Capture image, audio or video with consent.
2. Normalise and quality-check the input.
3. Apply OCR, ASR, object detection or segmentation where appropriate.
4. Fuse outputs with text instructions and retrieved context.
5. Generate an answer, structured decision or tool call.
6. Verify claims against sources or deterministic checks.
7. Present the result with uncertainty and provenance.

Multimodal systems introduce distinctive errors: visual hallucination, poor resolution handling, speaker misidentification, accent bias, temporal confusion and failures caused by missing context. Evaluation should therefore test modality-specific conditions such as lighting, noise, occlusion, handwriting, dialect, compression and adversarial content.

7. Illustrative Case: Multimodal Industrial Maintenance Assistant

Background

A manufacturing company maintains pumps and compressors across several sites. Technicians currently consult manuals, inspect equipment, take photographs and phone an expert when an alarm occurs.

Problem

The company wants faster diagnosis but cannot permit an AI system to authorise unsafe repairs or expose confidential plant information. Historical maintenance records are inconsistent, and photographs often contain irrelevant machinery.

Analysis

A general LLM lacks current equipment configuration and cannot reliably interpret every component. Fine-tuning alone would not solve the problem because manuals and maintenance status change. The dominant requirements are grounded retrieval, image interpretation, structured tool use and human approval.

Approach

The proposed system uses:

• A vision model to identify equipment labels, gauges and visible damage.
• OCR for serial numbers and displayed error codes.
• Retrieval over approved manuals, service bulletins and site-specific records.
• A language model adapted with PEFT for the company’s report format.
• Tools for checking inventory, work orders and equipment telemetry.
• A workflow graph that permits diagnosis suggestions but requires a supervisor to approve shutdowns or repairs.
• An audit record containing input images, retrieved sources, tool calls, model versions and approvals.

Synthetic images are generated to represent rare gauge positions and controlled damage patterns, but expert technicians validate the labels. Preference data is collected from technicians who rank explanations according to correctness, evidence use, clarity and safety—not merely confidence or brevity.

Outcome

In an evaluation pilot, the system is judged on diagnostic accuracy, evidence citation, unsafe-recommendation rate, time to draft a report and false escalation rate. It is deployed first in read-only mode. Only after monitoring demonstrates stable performance is it permitted to create draft work orders, still requiring human confirmation.

Learning points

The case illustrates that adaptation, retrieval, tools and multimodality solve different problems. Images supply observations, retrieval supplies authoritative context, tools supply live state, and human approval controls consequential action. The system’s reliability depends on the complete chain, not on the language model alone.

8. Governance, Ethics and Operational Best Practice

Adapted and multimodal systems may process personal data, biometric information, confidential documents or workplace recordings. Organisations should establish a lawful purpose, data minimisation, retention limits, access controls and deletion procedures. Consent is particularly important for voice, face and employee-monitoring applications.

Important governance practices include:

• Document data provenance, licences and annotator instructions.
• Maintain model cards, dataset documentation and change logs.
• Test demographic and linguistic performance differences.
• Separate development, evaluation and production data.
• Red-team prompt injection, data leakage and unsafe tool calls.
• Provide meaningful human oversight for high-impact decisions.
• Monitor drift, abuse, latency, cost and refusal behaviour.
• Define incident-response, rollback and decommissioning procedures.

Regulatory obligations vary by jurisdiction and application. A system used for employment, healthcare, education, finance or critical infrastructure may require risk assessment, transparency, record keeping and human review. Compliance should be designed into the architecture rather than added after deployment.

9. Contemporary Directions

Current research is moving toward smaller specialised models, mixture-of-experts systems, efficient long-context adaptation, multimodal foundation models, test-time reasoning, model routing and agent evaluation. Retrieval and tool use are increasingly combined with parameter-efficient adapters. Another important direction is verifiable generation: systems produce structured evidence, citations, program traces or executable checks rather than unsupported prose.

However, benchmark gains do not necessarily imply operational reliability. Future research must address compositional generalisation, calibrated uncertainty, long-horizon agent errors, multimodal grounding, energy use, copyright, privacy and evaluation contamination. The most valuable systems will be those whose claims and actions can be inspected, constrained and corrected.

Chapter Summary

Model adaptation specialises a general model for a task, domain or interaction style. Supervised fine-tuning changes model behaviour through labelled examples, while full fine-tuning is expensive and PEFT methods such as adapters and LoRA reduce trainable parameters and storage. Synthetic data expands coverage but requires provenance and independent validation. Preference optimisation improves behaviour according to rankings, although biased or superficial preferences can produce reward hacking and sycophancy.

Tool-using agents combine models with planning, typed tools, observations, memory, permissions and audit trails. Safe agents validate arguments, restrict privileges, control budgets and require human approval for consequential actions. Multimodal systems integrate text with images, speech, video and structured data through early, intermediate, late or tool-mediated fusion. Reliable deployment requires grounded evidence, modality-specific evaluation, privacy protection, monitoring and governance. No single adaptation method replaces careful system design.

Key Terms

Model adaptation: Methods for specialising a pretrained model for a target task or environment.

Supervised fine-tuning: Training a pretrained model on examples of desired inputs and outputs.

Parameter-efficient fine-tuning: Adaptation that updates only a small subset of parameters.

Adapter: A trainable module inserted into a frozen neural network.

LoRA: A low-rank parameterisation of a model-weight update.

QLoRA: LoRA training with a quantised frozen base model.

Synthetic data: Data generated by software, models, simulators or transformations.

Preference optimisation: Training that increases the likelihood of preferred responses.

RLHF: Reinforcement learning from human feedback.

Reference policy: A fixed model used as a comparison baseline in preference training.

Agent: A system that selects actions to pursue goals in an environment.

Tool calling: Structured invocation of an external function or service by a model.

ReAct: An agent pattern alternating actions and observations with reasoning-oriented steps.

Prompt injection: An attempt to manipulate model instructions through user or retrieved content.

Multimodal AI: AI that processes or generates multiple data modalities.

Early fusion: Combining modality representations near the beginning of processing.

Late fusion: Combining outputs from separately processed modalities.

Grounding: Connecting generated claims to observations, sources or tools.

Human-in-the-loop: Human oversight embedded in system decisions or actions.

Model drift: Performance change caused by changing data, users or environments.

Review Questions

Short Answer Questions

1. What is the main difference between full fine-tuning and PEFT?
2. Why can synthetic data amplify model errors?
3. What is the purpose of a reference policy in DPO?
4. Name four major components of a tool-using agent.
5. Distinguish early fusion from late fusion in multimodal AI.

Long Answer Questions

1. Compare supervised fine-tuning, retrieval augmentation and preference optimisation for an enterprise question-answering system.
2. Explain LoRA conceptually and discuss its operational advantages and limitations.
3. Describe a secure architecture for an agent that can read documents and update a business database.
4. Explain how speech, image and text processing can be combined in a customer-support application.
5. Discuss why benchmark performance is insufficient for evaluating a high-stakes generative-AI system.

Analytical / Application Questions

1. A company has current product information but inconsistent response formatting. Should it use retrieval, fine-tuning, PEFT or preference optimisation first? Justify your decision.
2. Design an approval policy for an agent that can inspect inventory, create purchase orders and cancel orders.
3. Develop an evaluation plan for a multimodal maintenance assistant, including accuracy, safety, fairness, latency and auditability measures.

Multiple Choice Questions with Answers

Question 1

Which method changes only a small subset of a model’s parameters while keeping the base model largely frozen?

A. Parameter-efficient fine-tuning

B. Random tokenisation

C. Unconstrained retrieval

D. Manual decoding

Correct Answer: A. Parameter-efficient fine-tuning

Question 2

In the LoRA expression W′ = W + BA, what does BA represent?

A. The frozen base model

B. The low-rank trainable update

C. The tokenisation vocabulary

D. The evaluation dataset

Correct Answer: B. The low-rank trainable update

Question 3

Why is retrieval usually more appropriate than fine-tuning for frequently changing factual information?

A. Retrieval can supply updated information at inference time

B. Fine-tuning cannot process text

C. Retrieval always eliminates hallucinations

D. Fine-tuning is never computationally possible

Correct Answer: A. Retrieval can supply updated information at inference time

Question 4

What is a major risk of using unverified synthetic training data?

A. It can reproduce and amplify generator errors

B. It always reduces training data size

C. It prevents any form of adaptation

D. It makes evaluation unnecessary

Correct Answer: A. It can reproduce and amplify generator errors

Question 5

In preference optimisation, what is the role of a preferred response?

A. It provides a response that training should favour relative to an alternative

B. It acts as an encryption key

C. It replaces all evaluation data

D. It specifies the model’s hardware

Correct Answer: A. It provides a response that training should favour relative to an alternative

Question 6

Which control is most appropriate before an agent executes an irreversible financial action?

A. Human confirmation and permission checking

B. Increasing response temperature

C. Removing the audit log

D. Allowing arbitrary generated code

Correct Answer: A. Human confirmation and permission checking

Question 7

What is indirect prompt injection?

A. Malicious instructions embedded in content that an agent reads

B. A hardware failure in a speech encoder

C. A method for compressing images

D. A type of database indexing

Correct Answer: A. Malicious instructions embedded in content that an agent reads

Question 8

Which fusion strategy combines decisions from separately processed modalities?

A. Late fusion

B. Early fusion

C. Token masking

D. Gradient clipping

Correct Answer: A. Late fusion

Question 9

Which metric is defined as TP ÷ (TP + FN)?

A. Recall

B. Precision

C. Latency

D. Perplexity

Correct Answer: A. Recall

Question 10

Why should a multimodal maintenance assistant begin in read-only mode?

A. To validate performance before permitting consequential actions

B. To prevent it from processing images

C. To guarantee that every diagnosis is correct

D. To eliminate the need for human expertise

Correct Answer: A. To validate performance before permitting consequential actions