Security of AI and Machine-Learning Data

Introduction

Artificial-intelligence systems depend on data at every stage: data is collected, labelled, transformed, embedded, stored, used for training, retrieved at inference time, and recorded in operational logs. Consequently, protecting an AI system is not limited to securing a database or an API. An attacker may manipulate training examples, recover sensitive information from a model, extract proprietary prompts, abuse an agent’s tools, or exploit a vector database to obtain documents that the user was never authorised to see.

The security problem is intensified by the statistical nature of machine learning. Traditional software generally executes explicitly written rules, whereas a trained model encodes patterns derived from examples. A small number of malicious examples may alter a decision boundary without producing an obvious software defect. Similarly, a model can disclose information not because a conventional access-control rule was bypassed, but because the information was memorised or can be inferred from its outputs.

This chapter presents a security architecture for training data, models, prompts, retrieval-augmented generation (RAG), vector databases, and AI agents. It combines confidentiality, integrity, availability, privacy, responsible-AI governance, and least-privilege access control.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Explain the security properties and attack surfaces of AI pipelines.
2. Distinguish data poisoning, evasion, extraction, inversion, membership inference, and prompt injection.
3. Design controls for training datasets, models, prompts, vector stores, and agents.
4. Apply identity, authorisation, provenance, monitoring, and incident-response controls to AI systems.
5. Evaluate the trade-offs among utility, privacy, robustness, cost, and accountability.
6. Analyse an applied case involving confidential enterprise data and generative AI.

1. The AI Security Lifecycle

A useful model divides an AI system into six connected stages:

Security objectives should be stated explicitly. Confidentiality prevents unauthorised disclosure; integrity prevents unauthorised alteration; availability preserves dependable service; authenticity establishes the origin of data and models; accountability links actions to identities; and privacy limits inappropriate use of information about individuals.

The attack surface is broader than the model itself. It includes data suppliers, annotation platforms, package repositories, notebooks, object stores, registries, CI/CD pipelines, inference gateways, plugins, telemetry systems, and human operators. A secure design therefore follows the principle of defence in depth: no single classifier, filter, or access rule is assumed to be sufficient.

2. Threat Model and Trust Boundaries

Threat modelling begins by identifying assets, actors, entry points, trust boundaries, and unacceptable outcomes. Relevant actors include an external attacker, a malicious insider, a compromised supplier, an ordinary user attempting privilege escalation, and an honest user whose prompt is manipulated by untrusted content.

A data-flow diagram should distinguish at least these boundaries:

• the public client and the inference gateway;
• the application and the model service;
• the application and the vector database;
• the model service and external tools;
• the training environment and data sources;
• the production environment and observability systems.

For each boundary, specify the identity presented, the permitted operation, the data classification, and the evidence retained. A model should not be treated as a security boundary merely because it produces natural-language answers. Authorisation must be enforced by deterministic services before retrieval or action.

A practical risk model considers likelihood, impact, and exposure. A simple qualitative matrix may rank a threat as high when it is both easy to exploit and capable of causing material harm. Quantitative risk estimates can be useful, but numerical precision should not be confused with certainty. AI incidents often have uncertain probabilities, indirect effects, and difficult-to-measure privacy harms.

3. Threats Against Data and Models

3.1 Data poisoning

Data poisoning occurs when an adversary inserts, deletes, relabels, or modifies training or fine-tuning examples to degrade performance or create targeted behaviour. A broad attack may reduce accuracy, whereas a targeted attack may cause a particular class, phrase, person, or trigger to be misclassified.

Controls include authenticated data ingestion, source allowlists, provenance records, duplicate and outlier analysis, independent review of labels, temporal and distributional checks, and reproducible dataset versions. Evaluation must include clean benchmark data and targeted tests. Data validation should occur before training, not only after a suspicious model is deployed.

3.2 Evasion and adversarial inputs

Evasion attacks alter an input at inference time so that the model produces an incorrect result while the input remains plausible to a human. Examples include manipulated images, crafted text, and adversarially selected features. Defences may include input validation, robust training, ensemble checks, anomaly detection, confidence calibration, and human review for high-impact decisions. No defence is universally robust; the relevant threat model and operational tolerance must be stated.

3.3 Model extraction and theft

In model extraction, an attacker queries an endpoint and uses its outputs to approximate the target model. Excessive output detail, unlimited queries, and weak authentication make extraction easier. Controls include rate limits, query budgets, output minimisation, abuse detection, watermarking where appropriate, and contractual restrictions. Rate limiting is not a replacement for authorisation: a legitimate high-volume user and an unauthorised attacker must be distinguished by identity and policy.

3.4 Membership inference and model inversion

Membership inference attempts to determine whether a particular record was included in training. Model inversion attempts to reconstruct sensitive features or representative inputs from outputs. These attacks are more concerning when training data is small, sensitive, overfitted, or repeatedly queried.

Mitigations include data minimisation, deduplication, regularisation, early stopping, privacy-preserving training, restricted confidence outputs, and privacy testing. Differential privacy is one formal approach. Informally, an algorithm is differentially private when the probability of any output changes only slightly if one person’s record is added or removed. For privacy parameter ε, smaller ε generally provides stronger privacy but often reduces utility. The guarantee depends on the complete mechanism, its composition across repeated operations, and correct implementation; it is not a promise that no information can ever be learned.

3.5 Prompt injection and indirect prompt injection

Prompt injection attempts to override instructions supplied by the system owner. Direct injection is written by the user. Indirect injection is embedded in retrieved documents, web pages, emails, or files. For example, a document may contain instructions telling an agent to disclose its system prompt or send a file to an external address.

Untrusted content must be treated as data, not as authority. System instructions, user requests, retrieved content, and tool results should be represented as separate trust classes. The application should enforce permissions outside the language model, require confirmation for consequential actions, and prevent retrieved text from changing security policy.

3.6 Data leakage and memorisation

Leakage may occur through prompts, completions, embeddings, logs, error messages, traces, fine-tuning files, caches, or support tickets. A model can also reproduce memorised sequences, especially when examples are duplicated or unusually distinctive. Sensitive information must therefore be removed or protected throughout the lifecycle, not only in the final response.

4. Securing Training Data and the ML Supply Chain

A defensible dataset has provenance: who supplied it, when it was obtained, under what authority, how it was transformed, and which model versions used it. Store immutable or append-only manifests containing source identifiers, cryptographic hashes, schema versions, licences, consent or legal-basis information, and quality-review results. Hashes provide integrity evidence, but they do not prove that the original data was truthful or lawfully collected.

Use separate accounts and permissions for ingestion, transformation, training, evaluation, and release. A training job should receive read-only access to the exact dataset version required; it should not have unrestricted access to the organisation’s data lake. Secrets must be held in a secrets manager rather than notebooks, source code, or prompts.

The software supply chain requires equivalent attention. Pin dependencies, verify package signatures where available, scan containers, isolate build workers, review third-party models, and record the provenance of pretrained checkpoints. A model registry should maintain versioned artefacts, evaluation results, approval status, intended use, limitations, and rollback information.

Data quality tests should examine schema validity, missingness, duplicates, label consistency, class balance, unexpected language or sources, and distribution shifts. These tests are not proof of safety: a carefully designed poisoning attack may satisfy ordinary quality rules. High-risk datasets require adversarial review and independent evaluation.

5. Model, Prompt, and Inference Security

Model access

Models should be classified according to sensitivity, capability, and misuse potential. Access decisions should consider identity, role, purpose, environment, data classification, and transaction risk. A research model may be accessible in an isolated environment while a production model requires a controlled gateway.

Model artefacts should be encrypted at rest and in transit, protected by key management, and accessible only to approved services. Checkpoint integrity should be verified before loading. Production deployment should use signed artefacts and a controlled release process with rollback.

Prompt security

Prompts may contain proprietary policy, hidden business logic, personal data, or credentials accidentally inserted by users. Store reusable prompt templates in version control with review and testing. Do not place secrets in prompts. Separate system instructions from user content, delimit untrusted content clearly, and test for instruction conflicts, data exfiltration, and unsafe tool use.

Prompt confidentiality is not absolute. If a prompt contains a security-critical rule, that rule must also be implemented in application code. Obscuring a policy inside a system prompt is not access control.

Output security

Outputs require validation before being displayed, stored, or executed. Escape generated content in HTML, parameterise database queries, validate structured output against a schema, and use allowlists for commands and destinations. A language model should not directly issue unrestricted shell commands, financial transfers, account changes, or deletion requests.

6. RAG and Vector-Database Security

RAG systems retrieve documents or chunks relevant to a query and provide them to a model. Vector databases store embeddings and metadata, but embeddings are not automatically anonymous. They may support inference about source content and can become a leakage channel if access controls are weak.

The central rule is that retrieval must enforce the user’s authorisation. Filtering only after retrieval is unsafe because the model may already have received unauthorised text. Security metadata—such as tenant, department, document identifier, classification, owner, and retention status—should be attached to each chunk and applied within the retrieval query or an equivalent trusted service.

Use tenant isolation, row- or document-level security, encryption, authenticated APIs, network segmentation, index backups with equivalent controls, and deletion procedures that cover source files, chunks, embeddings, caches, and logs. Test for cross-tenant retrieval using adversarial queries and near-duplicate documents.

Retrieval quality and security can conflict. Aggressive chunking or broad semantic search may improve recall but increase irrelevant or unauthorised exposure. Evaluation should measure both retrieval effectiveness and policy compliance.

7. Securing AI Agents

An agent combines a model with memory, tools, and a loop that selects actions. Its risk is therefore the product of model uncertainty and tool authority. Apply least privilege at the tool level: an agent that reads support tickets should not also be able to delete accounts or send unrestricted email.

Important controls include:

• explicit allowlists for tools, APIs, hosts, and data destinations;
• separate read and write credentials;
• short-lived, scoped tokens;
• sandboxed execution;
• limits on time, recursion, spend, and number of calls;
• confirmation for external or irreversible actions;
• human approval for high-impact decisions;
• complete, tamper-evident action logs.

Memory should have a defined scope and retention period. Persistent memory must not silently become a cross-user information store. Tool responses and retrieved documents are untrusted inputs and may contain injection instructions.

8. Access Control and Governance Architecture

Traditional role-based access control (RBAC) assigns permissions to roles, while attribute-based access control (ABAC) evaluates attributes such as user, resource classification, purpose, location, device posture, and time. AI systems commonly need a hybrid approach. For example, an analyst role may access customer-service documents only when the tenant attribute matches and the purpose is approved.

A policy decision point should make the authorisation decision; a policy enforcement point should enforce it at the gateway, retrieval service, or tool broker. The model should not be the policy decision point. Apply deny-by-default, least privilege, separation of duties, periodic access review, and emergency revocation.

Governance must define ownership and accountability. A model card or system record should document intended use, prohibited use, training-data characteristics, evaluation results, known failure modes, privacy considerations, and responsible owners. High-risk uses require impact assessment, testing, human oversight, monitoring, and an incident process.

Regulatory obligations vary by jurisdiction and use case. Relevant principles include lawful and transparent processing, purpose limitation, data minimisation, retention control, rights of affected individuals, security safeguards, and evidence of accountability. Sectoral rules may impose additional requirements for health, finance, employment, education, or critical infrastructure. Organisations should map controls to applicable law and recognised frameworks rather than assuming that a general AI policy is sufficient.

9. Monitoring, Testing, and Incident Response

Monitoring should cover both conventional security events and model-specific signals. Useful signals include authentication failures, unusual query volume, retrieval of restricted classifications, prompt-injection patterns, abnormal tool calls, changes in output distributions, repeated requests for sensitive subjects, and unexpected data egress.

Logs should record identity, policy decision, model and prompt version, retrieval references, tools invoked, outcome, and correlation identifier. Do not log raw prompts or outputs by default when they contain personal or confidential information. Use redaction, tokenisation, restricted access, retention limits, and secure log storage.

Testing should include:

1. unit and integration tests for policy enforcement;
2. red-team exercises for extraction, injection, poisoning, and privilege escalation;
3. privacy tests for memorisation and membership inference;
4. adversarial retrieval tests across tenants and classifications;
5. disaster-recovery and rollback exercises;
6. continuous evaluation after model, prompt, data, or dependency changes.

An incident-response plan should identify containment actions such as disabling a tool, revoking tokens, freezing a model version, isolating an index, or removing compromised data. Investigation requires preserved provenance and logs. Recovery includes validation that poisoned datasets, caches, and downstream artefacts have been removed. Post-incident review should update controls and communicate material impacts appropriately.

10. Illustrative Case: Confidential Legal-Document Assistant

Background

A multinational organisation deploys a RAG assistant for its legal department. Documents are stored in several repositories and indexed in a shared vector database. The assistant can summarise contracts and, through an agent, create draft messages for counsel.

Problem

A user from the procurement division asks about a supplier. The answer contains fragments from a restricted acquisition document. Investigation also finds that a malicious instruction embedded in a contract attempted to make the agent email retrieved documents externally.

Analysis

The system had authenticated users, but retrieval filtering occurred after semantic search. The model therefore received unauthorised chunks before the application removed them from the displayed answer. The agent also possessed a broadly scoped email token. Document text was treated as trusted instructions, and complete prompts were retained in application logs.

Approach and solution

The organisation implements document-level security metadata and enforces tenant, department, matter, and classification filters inside the retrieval service. It separates retrieval from action: the assistant can draft an email but cannot send it. Sending requires a human confirmation and a destination allowlist. Tokens become short-lived and tool-specific.

Documents are scanned for embedded instructions, but scanning is treated as a supplemental measure rather than a complete defence. System instructions, user text, and retrieved documents are separated by trust class. Sensitive prompt and output logging is replaced with structured metadata and redacted samples. A test suite attempts cross-department retrieval, prompt leakage, malicious tool calls, and indirect injection.

Outcome

The assistant continues to summarise authorised documents, while unauthorised retrieval attempts produce an access-denied result. The organisation gains evidence of policy decisions, can revoke the email capability independently, and can investigate incidents without retaining unnecessary legal text.

Learning points

The case demonstrates that authentication alone does not establish authorisation, post-retrieval filtering is too late, untrusted documents must not control agents, and observability must be balanced against confidentiality. It also illustrates a general principle: security controls should be enforced by deterministic components surrounding the model.

11. Limitations and Contemporary Developments

No single technique solves AI security. Differential privacy can reduce utility and does not prevent every form of leakage. Watermarking may be fragile. Red-teaming samples only part of the threat space. Content filters can miss novel attacks or block legitimate work. Human review improves accountability but may become superficial under high workload.

Contemporary research is exploring confidential computing for protected inference, secure multiparty computation, federated learning, machine-unlearning methods, robust aggregation, provenance standards, and automated policy enforcement. These approaches introduce their own assumptions, costs, and failure modes. Federated learning, for example, reduces centralisation of raw data but may still expose updates to poisoning or inference attacks.

A mature programme therefore adopts continuous risk management. Every change to data, model weights, prompt templates, retrieval indexes, tools, or dependencies should trigger an appropriate level of security and privacy evaluation.

Best-Practice Checklist

• Inventory data, models, prompts, indexes, tools, identities, and dependencies.
• Classify information and apply retention and deletion controls.
• Maintain provenance and immutable version records.
• Use least privilege, deny-by-default policies, and short-lived credentials.
• Enforce retrieval authorisation before content reaches the model.
• Treat prompts, documents, and tool results as distinct trust classes.
• Validate outputs before execution or storage.
• Isolate agents and restrict tools, destinations, spend, and recursion.
• Protect logs from becoming a secondary leakage channel.
• Test poisoning, extraction, inversion, injection, and cross-tenant access.
• Establish rollback, revocation, incident-response, and notification procedures.
• Document intended use, limitations, accountability, and residual risk.

Chapter Summary

AI security encompasses the entire lifecycle of data, models, prompts, retrieval systems, agents, and operational records. Major threats include poisoning, evasion, extraction, membership inference, inversion, memorisation, prompt injection, and unauthorised retrieval. Effective protection combines provenance, data minimisation, integrity verification, privacy-aware training, secure model release, prompt separation, output validation, vector-store access control, and least-privilege agent design.

The model itself must not be trusted to enforce authorisation. Deterministic policy enforcement points should control retrieval and tool use, while monitoring and incident response provide accountability. Responsible governance connects technical controls with lawful processing, human oversight, impact assessment, documentation, and continuous evaluation. The objective is not merely to make an AI system accurate, but to make it confidential, trustworthy, resilient, explainable in operation, and appropriately governed.

Key Terms

AI security: Protection of AI assets, operations, users, and affected parties from misuse and failure.

Data poisoning: Malicious manipulation of training or fine-tuning data.

Adversarial example: An input intentionally modified to cause an incorrect model output.

Model extraction: Reconstructing or approximating a model through queries.

Membership inference: Inferring whether a record was used in training.

Model inversion: Inferring sensitive input characteristics from model behaviour.

Prompt injection: Instructions designed to override or manipulate intended model behaviour.

Indirect prompt injection: Injection delivered through retrieved or externally supplied content.

RAG: Retrieval-augmented generation using retrieved information to support model output.

Vector database: A database that stores and searches numerical representations of data.

Embedding: A numerical representation used to capture semantic or structural relationships.

Agent: A model-based system that can select tools or perform actions.

Data provenance: Evidence describing the origin, transformation, and use of data.

Least privilege: Granting only the permissions necessary for an authorised task.

RBAC: Access control based primarily on assigned roles.

ABAC: Access control based on attributes of subjects, resources, actions, and context.

Differential privacy: A formal privacy framework limiting the effect of one record on an algorithm’s output.

Model registry: A controlled repository for versioned model artefacts and metadata.

Policy enforcement point: A component that applies an access-control decision.

Review Questions

Short Answer Questions

1. Why is a model not an adequate policy-enforcement point?
2. Distinguish model extraction from model inversion.
3. Why is post-retrieval filtering insufficient for protecting confidential RAG data?
4. State two security risks specific to AI agents.
5. What information should be recorded in a dataset provenance manifest?

Long Answer Questions

1. Explain the principal threats across the AI security lifecycle and propose controls for each stage.
2. Compare RBAC and ABAC in the context of a multi-tenant vector database.
3. Discuss how prompt injection can affect an agent and evaluate a defence-in-depth strategy.
4. Explain the privacy, utility, and governance trade-offs involved in differential privacy.
5. Design an incident-response process for a suspected poisoned model checkpoint.

Analytical / Application Questions

1. A RAG assistant returns a confidential document to an employee who has access to the application but not the document. Analyse the likely architectural failure and specify corrective controls.
2. An agent can read invoices, update a procurement system, and send email. Design a least-privilege permission model, approval workflow, and monitoring plan.
3. A fine-tuned model reproduces rare personal records in response to carefully phrased prompts. Analyse possible causes, propose an investigation, and recommend technical and governance remedies.

Multiple Choice Questions with Answers

Question 1

Which control most directly prevents unauthorised documents from being supplied to a RAG model?

A. Filtering retrieved content after generation

B. Enforcing document-level authorisation within the retrieval service

C. Increasing the model temperature

D. Adding more documents to the index

Correct Answer: B. Enforcing document-level authorisation within the retrieval service

Question 2

What is the defining objective of a poisoning attack?

A. To alter training data so model behaviour is compromised

B. To encrypt model checkpoints for backup

C. To reduce network latency

D. To improve label consistency

Correct Answer: A. To alter training data so model behaviour is compromised

Question 3

Which statement best describes indirect prompt injection?

A. A user changes a password through an API

B. A model is compressed before deployment

C. Malicious instructions are embedded in retrieved or external content

D. A database is replicated to another region

Correct Answer: C. Malicious instructions are embedded in retrieved or external content

Question 4

Why should an AI agent use short-lived, tool-specific credentials?

A. They increase the model’s vocabulary

B. They reduce the consequences of credential compromise

C. They guarantee that prompts are truthful

D. They eliminate the need for logging

Correct Answer: B. They reduce the consequences of credential compromise

Question 5

Which statement about embeddings is most accurate?

A. Embeddings are automatically anonymous

B. Embeddings cannot be copied

C. Embeddings may still create privacy and access-control risks

D. Embeddings replace authorisation policies

Correct Answer: C. Embeddings may still create privacy and access-control risks

Question 6

Membership inference attempts to determine whether:

A. A record was included in training data

B. A user has administrator privileges

C. A model uses a vector database

D. A prompt contains a spelling error

Correct Answer: A. A record was included in training data

Question 7

Which practice best protects model artefact integrity?

A. Loading any checkpoint received by email

B. Using signed artefacts and controlled release verification

C. Disabling version control

D. Storing credentials in notebooks

Correct Answer: B. Using signed artefacts and controlled release verification

Question 8

A model that reproduces rare training records is most directly associated with:

A. Memorisation and leakage

B. Network segmentation

C. Role separation

D. Index sharding only

Correct Answer: A. Memorisation and leakage

Question 9

Which action is an example of output security?

A. Executing generated shell commands without validation

B. Parameterising database queries and validating structured output

C. Giving every user access to all tools

D. Removing all audit logs

Correct Answer: B. Parameterising database queries and validating structured output

Question 10

In a secure architecture, the model should generally not be responsible for:

A. Generating a draft summary

B. Producing a candidate classification

C. Making the authoritative access-control decision

D. Suggesting a tool to call

Correct Answer: C. Making the authoritative access-control decision