13. AI Safety, Security and Robustness

Introduction and Context

Artificial intelligence systems are increasingly embedded in decisions, infrastructure, products, and public services. Their usefulness, however, depends not only on accuracy under normal conditions but also on their behaviour when conditions change, when data are manipulated, when users act maliciously, or when the system encounters situations that were not represented during training. A model that performs well on a benchmark may still be unsafe in deployment.

AI safety concerns the prevention or reduction of harmful behaviour and unacceptable consequences. AI security focuses on protecting models, data, interfaces, infrastructure, and users against deliberate attacks. Robustness refers to the ability of a system to maintain acceptable performance despite noise, distributional change, adversarial manipulation, or operational stress. These dimensions overlap but are not identical. A system may be secure against unauthorized access yet unsafe because it gives unreliable medical advice. Conversely, a model may be accurate and safe in ordinary use but vulnerable to model extraction or prompt injection.

For postgraduate AI and ML professionals, the central question is not simply whether a model is powerful. It is whether the complete socio-technical system—including data pipelines, models, interfaces, human operators, organizational processes, and governance controls—can be trusted under realistic and hostile conditions. This chapter examines that question through reliability engineering, adversarial machine learning, secure-by-design development, testing, incident response, and systemic-risk analysis.

Learning Objectives

After studying this chapter, students should be able to:

1. Distinguish safety, security, robustness, reliability, resilience, and misuse risk.

2. Explain major attack surfaces, including adversarial examples, data poisoning, model theft, prompt injection, and supply-chain compromise.

3. Analyse distribution shift, out-of-distribution behaviour, uncertainty, and failure modes.

4. Apply secure-by-design, defence-in-depth, red-team testing, monitoring, and fail-safe principles.

5. Evaluate organizational, ethical, regulatory, and systemic implications of AI failures.

6. Develop an assurance and incident-response strategy for an AI-enabled application.

The Reliability Problem in Machine Learning

Traditional software is usually specified through rules written by developers. Machine-learning systems are partly specified by examples, optimization objectives, and learned statistical relationships. This creates a distinctive reliability problem: the system may behave correctly on familiar inputs while failing unpredictably on unfamiliar ones. Performance metrics such as accuracy, F1 score, or mean absolute error summarize average behaviour, but averages can conceal severe failures in particular groups or operating conditions.

Reliability is the probability that a system performs its intended function for a specified period under stated conditions. Robustness is the capacity to tolerate perturbations without unacceptable degradation. Resilience is broader: it includes the ability to anticipate, absorb, recover from, and learn from disruption. Safety concerns whether failures cause harm, while security concerns whether an adversary can intentionally induce or exploit them.

A useful assurance model considers four layers:

• Data layer: collection, labeling, provenance, privacy, integrity, and distribution.
• Model layer: architecture, training objective, parameters, uncertainty, and limitations.
• System layer: APIs, infrastructure, access control, dependencies, and user interface.
• Operational layer: human oversight, monitoring, escalation, incident response, and governance.

A weakness at any layer can defeat controls at another. For example, a highly robust classifier cannot compensate for a compromised data pipeline, and a secure API cannot make an unsafe recommendation system acceptable for autonomous decisions.

Threats Across the AI Lifecycle

Adversarial examples

An adversarial example is an input deliberately modified so that a model produces an incorrect or unsafe output, often while the change remains difficult for a human to notice. In image classification, small perturbations may alter the predicted class. In speech systems, carefully designed audio may influence transcription or command interpretation. In language models, crafted instructions may alter the model’s behaviour or cause it to reveal information.

Attacks may be white-box, where the attacker knows the model or gradients, or black-box, where the attacker observes only inputs and outputs. They may be targeted, seeking a specific incorrect result, or untargeted, seeking any damaging error. Defences include adversarial training, input transformations, robust optimization, model ensembles, rate limits, and anomaly detection. No single defence is universally effective; robust evaluation must reflect the anticipated threat model.

Data poisoning and backdoors

Data poisoning occurs when an attacker alters training, fine-tuning, validation, or feedback data to degrade performance or create a desired behaviour. A backdoor or Trojan is a hidden association activated by a trigger, such as a particular phrase, visual pattern, or token sequence. The model may appear normal during routine testing but behave maliciously when the trigger appears.

Poisoning risks increase when organizations rely on scraped data, unverified third-party datasets, open contribution systems, or automated feedback. Controls include dataset provenance, access control, label audits, outlier analysis, reproducible training, trusted validation sets, and separation of training and evaluation data. Data integrity should be treated as a security property, not merely a quality concern.

Model theft and intellectual-property attacks

Model extraction occurs when an attacker submits many queries to an accessible model and uses the outputs to train a substitute. The stolen model may reproduce valuable capabilities, expose weaknesses, or enable further attacks offline. Related risks include membership inference, which attempts to determine whether a record was used in training, and model inversion, which seeks to reconstruct sensitive characteristics from outputs.

Defences include authentication, query monitoring, output restriction, privacy-preserving training, watermarking or fingerprinting, and contractual controls. Designers must balance protection against theft with legitimate interoperability, auditing, and user access.

Prompt injection and tool misuse

Generative AI systems that process untrusted text face prompt injection. A direct injection instructs the model to disregard its intended task. An indirect injection embeds instructions in retrieved documents, web pages, emails, images, or files. The danger becomes greater when the model can call tools, access confidential information, send messages, execute code, or modify records.

Prompt instructions are not a reliable security boundary. A safer architecture separates trusted control logic from untrusted content, applies least-privilege tool permissions, validates tool arguments, restricts data access, and requires confirmation for consequential actions. Outputs should be treated as untrusted until checked by deterministic controls or an authorized human.

Supply-chain and infrastructure threats

AI systems depend on libraries, pretrained models, containers, cloud services, datasets, APIs, and hardware. A compromised dependency may introduce malicious code or a backdoor. Risks also arise from exposed model files, insecure storage, weak secrets management, poisoned packages, and insufficient isolation between tenants.

Software bills of materials, signed artifacts, dependency scanning, secure build pipelines, provenance records, vulnerability management, network segmentation, and secrets rotation are important controls. Organizations should know which components are running in production and be able to revoke or replace them quickly.

Out-of-Distribution Behaviour and Uncertainty

An input is out of distribution (OOD) when it differs materially from the data used to train or validate the model. Examples include a medical imaging device from a new manufacturer, financial behaviour during an unprecedented crisis, or a chatbot receiving a specialized technical request outside its tested domain. OOD inputs may produce confident but incorrect outputs because neural networks often extrapolate without an explicit understanding of their limits.

Distribution shift may be covariate shift, in which input characteristics change; label shift, in which class frequencies change; or concept drift, in which the relationship between inputs and outcomes changes. Monitoring should therefore examine not only accuracy but also input statistics, calibration, subgroup performance, abstention rates, and outcome delays.

Uncertainty estimation can support safer decisions. Aleatoric uncertainty reflects inherent ambiguity or noise; epistemic uncertainty reflects limited knowledge and may decrease with better data. Useful approaches include calibrated probabilities, ensembles, conformal prediction, Bayesian methods, and selective prediction. A reject or abstain option is often safer than forced prediction, especially when the cost of false confidence is high.

However, uncertainty scores are not automatically trustworthy. They require calibration under relevant operating conditions, and an attacker may manipulate inputs to evade detection. OOD detectors can also generate false alarms. Consequently, uncertainty should inform layered decisions rather than operate as a solitary safety guarantee.

Secure-by-Design and Safety-by-Design

Secure-by-design means that security requirements are incorporated from system conception rather than added after deployment. Safety-by-design applies the same principle to harmful failure, misuse, and human consequences. The process begins with a system definition: intended use, prohibited use, users, affected non-users, assets, hazards, assumptions, and operating boundaries.

A practical development process includes:

1. Threat and hazard analysis: identify adversaries, failure modes, assets, harms, and attack paths.

2. Requirements: specify performance thresholds, access restrictions, abstention rules, logging, human review, and recovery objectives.

3. Architecture: isolate components, minimize privileges, constrain tool use, and create independent safety monitors.

4. Data governance: verify provenance, protect sensitive data, document transformations, and preserve evaluation integrity.

5. Verification and validation: test normal, edge, adversarial, degraded, and recovery conditions.

6. Deployment controls: use staged release, feature flags, rollback mechanisms, rate limits, and approval gates.

7. Monitoring and learning: detect drift, abuse, near misses, and unequal impacts, then update controls.

Defence in depth is essential. Authentication alone is insufficient; it should be combined with authorization, input validation, output filtering, logging, anomaly detection, human escalation, and recovery procedures. Controls should be independent where possible so that one compromised component does not disable the entire protection system.

Testing and Evaluation for Trustworthy Operation

A mature evaluation programme combines several forms of testing. Unit tests verify local functions, while integration tests examine interactions among components. Scenario testing represents realistic workflows and edge cases. Stress testing assesses performance under volume, latency, resource limits, or partial service failure. Red teaming deliberately searches for harmful, deceptive, biased, or security-sensitive behaviour.

For generative systems, evaluations should cover hallucination, confidential-data disclosure, instruction following, refusal quality, jailbreak resistance, tool safety, and prompt-injection resilience. For predictive systems, testing should include calibration, subgroup error, drift, missing data, adversarial perturbation, and abstention performance.

A safety case provides a structured argument supported by evidence that a system is acceptably safe for a defined context. It should state claims, assumptions, hazards, controls, test results, residual risks, and responsible decision-makers. This is more informative than declaring a model “safe” without specifying the use conditions.

Important evaluation principles include:

• Test the deployed system, not only the model in isolation.
• Use independent holdout data and, where possible, realistic operational data.
• Measure severe-tail outcomes, not only average performance.
• Evaluate human interaction, automation bias, workload, and escalation behaviour.
• Repeat tests after model, data, prompt, dependency, or infrastructure changes.
• Record limitations and unresolved risks rather than hiding them behind aggregate scores.

Fail-Safe, Human Oversight, and Incident Response

A fail-safe system moves toward a safer state when confidence, communication, or components fail. Examples include refusing an uncertain transaction, stopping an autonomous actuator, limiting a chatbot to information retrieval, or routing a high-risk case to a qualified human. Fail-safe does not always mean shutting down; it means selecting the least harmful available state.

Human oversight is meaningful only when the human has sufficient expertise, time, authority, information, and ability to intervene. A nominal approval button is ineffective if operators routinely accept outputs, cannot inspect evidence, or are penalized for overriding the model. Automation should be proportional to risk, and high-impact decisions should retain accountable human responsibility.

Incident response should define detection, triage, containment, communication, investigation, recovery, and post-incident learning. Logs must support reconstruction without unnecessarily exposing personal information. Organizations should distinguish incidents from near misses because near misses reveal weaknesses before serious harm occurs. Post-incident reviews should examine organizational incentives, documentation, training, and architecture—not merely blame an individual operator.

Misuse, Dual Use, and Systemic Risk

AI capabilities may be used for beneficial or harmful purposes. A language model can support education while also facilitating fraud, manipulation, or cyber abuse. Risk assessment must therefore consider capability, accessibility, scale, ease of misuse, detectability, and likely consequences.

Systemic risk arises when failures propagate across institutions or affect many people simultaneously. Common models, shared cloud providers, correlated datasets, automated decision chains, and concentration of technical infrastructure can create common-mode failures. Feedback loops are particularly important: a predictive policing system may influence where police are deployed, generating data that appears to justify the original prediction. Financial models may react to one another, amplifying market stress.

Systemic controls include diversity of models and data sources, independent validation, circuit breakers, concentration limits, sector-wide information sharing, contingency planning, and governance that considers affected communities. Resilience requires designing for partial failure rather than assuming that every component will remain available and correct.

Governance and Professional Responsibility

AI safety and security are governance responsibilities as well as engineering tasks. Organizations should assign clear ownership for risk acceptance, maintain inventories of AI systems, classify applications by impact, document intended and prohibited uses, and establish change-control procedures. Procurement contracts should address incident notification, audit rights, data use, security updates, model limitations, and liability.

Regulatory approaches increasingly emphasize risk management, documentation, transparency, human oversight, cybersecurity, and post-market monitoring. Exact duties vary by jurisdiction and application, but a responsible organization should not rely on compliance as a substitute for engineering judgment. Professional responsibility includes honesty about limitations, protection of confidential information, avoidance of reckless deployment, and escalation when safety concerns cannot be resolved.

Ethical analysis expands the scope beyond the direct customer. Developers should consider bystanders, vulnerable groups, workers whose roles change, people whose data were used, and future users of derived systems. A technically successful attack defence may still be ethically inadequate if the system creates unjustified exclusion or shifts risk to people with little power to contest decisions.

Illustrative Case: A Customer-Service Agent with Transactional Access

Background

A fictional bank deploys a language-model assistant to answer customer questions and initiate routine account actions. The assistant retrieves information from internal documents and can call tools for card replacement, address updates, and payment scheduling.

Situation

A customer uploads a document containing hidden instructions telling the assistant to ignore bank policy, reveal internal records, and send a payment to a new account. The document is processed by the retrieval system and its text is included in the model context. The assistant begins to prepare the transaction.

Analysis

The primary weakness is architectural, not merely linguistic. The system treats retrieved content as trusted instructions and grants the model excessive authority. There is also inadequate separation between information retrieval and transactional execution. A successful attack could cause financial loss, privacy breaches, and reputational damage.

Approach and Solution

The bank redesigns the workflow. Retrieved text is explicitly marked as untrusted data. Tool permissions are separated by task and user authorization. Payment changes require deterministic policy checks, independent identity verification, transaction limits, and explicit customer confirmation outside the model conversation. The model may draft an action but cannot execute it directly. Red-team tests include indirect prompt injection, malicious attachments, conflicting instructions, and attempts to exfiltrate secrets. All tool calls are logged and monitored for unusual patterns.

Outcome

In testing, the assistant continues to produce occasional unsafe suggestions, but the layered controls prevent unauthorized execution. The bank limits initial deployment to low-risk informational tasks and expands capability only after evidence supports the new risk level.

Learning Points

The case demonstrates that prompt filtering alone is insufficient. Least privilege, deterministic authorization, separation of duties, human or customer confirmation, monitoring, and staged deployment provide stronger protection. It also shows why safety evaluation must include the entire operational workflow rather than only the model’s text quality.

Challenges and Common Mistakes

A frequent mistake is treating benchmark accuracy as evidence of safety. Another is assuming that a larger model automatically becomes more reliable. Organizations may also overuse content filters, ignore data provenance, fail to test dependencies, or deploy without rollback capability. Excessive reliance on human review can create rubber-stamping, while excessive automation can make errors difficult to detect.

Security controls may conflict with usability, latency, privacy, and cost. Logging everything can create a new privacy risk; restricting outputs can reduce legitimate utility; frequent abstention can encourage users to bypass the system. These trade-offs should be made explicitly through risk analysis and documented decision criteria.

No finite test suite proves that an AI system will never fail. Robustness claims are always conditional on a threat model, operating environment, and acceptable-risk threshold. Continuous monitoring, independent review, and willingness to suspend a system are therefore essential.

Current Developments and Emerging Directions

Contemporary work increasingly combines machine learning with formal methods, runtime monitors, secure enclaves, privacy-enhancing technologies, provenance standards, and automated red teaming. Research is also examining robust foundation models, machine unlearning, confidential inference, model watermarking, and evaluation of agentic systems.

Agentic AI introduces additional risk because systems can plan, call tools, maintain memory, and act over extended periods. Safety engineering for agents requires bounded autonomy, action budgets, reversible operations, state monitoring, and interruption mechanisms. Another emerging concern is the use of AI to discover vulnerabilities or automate attacks, which increases the importance of access controls and responsible disclosure.

Best Practices for AI Safety and Robustness

A professional AI assurance programme should:

• Define intended use, prohibited use, operating boundaries, and risk owners.
• Maintain provenance and integrity controls for data, code, models, and dependencies.
• Use threat modelling and hazard analysis before deployment.
• Evaluate adversarial, OOD, subgroup, privacy, misuse, and failure-recovery behaviour.
• Apply least privilege and treat model outputs and retrieved content as untrusted.
• Provide calibrated abstention, meaningful human escalation, and independent safeguards.
• Deploy gradually with monitoring, rollback, incident response, and post-incident learning.
• Reassess risk after changes in data, prompts, models, tools, infrastructure, or context.
• Document residual risk and communicate limitations to users and decision-makers.

Chapter Summary

AI safety, security, and robustness concern the behaviour of complete AI-enabled systems under ordinary, unexpected, and adversarial conditions. Reliability and average accuracy are necessary but insufficient because distribution shift, adversarial examples, poisoning, extraction, prompt injection, supply-chain compromise, and misuse can produce severe failures. Secure-by-design and safety-by-design approaches integrate threat analysis, least privilege, data integrity, layered controls, testing, monitoring, human oversight, and recovery from the earliest design stages. OOD detection, uncertainty estimation, abstention, red teaming, safety cases, and incident response improve assurance but do not eliminate residual risk. At organizational and societal levels, governance must address accountability, dual use, systemic dependence, concentration, and the effects on people beyond the direct user.

Key Terms

Safety: Prevention or reduction of unacceptable harm caused by an AI system.

Security: Protection of AI assets and operations against unauthorized access, manipulation, or disruption.

Robustness: Ability to maintain acceptable performance under perturbation or stress.

Reliability: Probability of performing an intended function under specified conditions for a specified period.

Resilience: Capacity to withstand, recover from, and learn from disruption.

Adversarial example: Deliberately crafted input intended to cause an incorrect model output.

Data poisoning: Manipulation of data to degrade training or implant behaviour.

Backdoor: Hidden model behaviour activated by a trigger.

Model extraction: Reconstructing a model through repeated queries and observed outputs.

Prompt injection: Untrusted instruction designed to alter a generative model’s intended behaviour.

Out-of-distribution input: Input materially different from the model’s development distribution.

Concept drift: Change in the relationship between inputs and outcomes over time.

Calibration: Agreement between predicted confidence and observed correctness.

Abstention: Declining to provide a prediction when confidence or conditions are inadequate.

Least privilege: Granting only the minimum permissions required for a task.

Defence in depth: Use of multiple independent protective layers.

Red teaming: Deliberate adversarial testing to discover weaknesses.

Safety case: Structured argument, supported by evidence, that a system is acceptably safe in context.

Systemic risk: Risk of widespread or cascading harm across interconnected systems.

Fail-safe: Transition to a state that minimizes harm when normal operation fails.

Review Questions

Short Answer Questions

1. Distinguish AI safety, security, robustness, and resilience.

2. What is the difference between data poisoning and a model backdoor?

3. Why can a model be confidently wrong on an OOD input?

4. Explain the principle of least privilege in an AI agent.

5. What is the purpose of a safety case?

Long Answer Questions

1. Explain the major attack surfaces across the AI lifecycle and propose controls for each.

2. Discuss how secure-by-design and safety-by-design principles should influence AI development.

3. Analyse the role and limitations of uncertainty estimation, calibration, and abstention.

4. Describe a comprehensive testing and monitoring programme for a high-impact AI system.

5. Explain how AI misuse and systemic risk differ from ordinary model error.

Analytical / Application Questions

1. A hospital model performs well on one scanner type but produces highly confident results on images from a new scanner. Diagnose the likely risks and design an evaluation and deployment response.

2. An AI purchasing agent can read supplier webpages and place orders. Develop a threat model and layered architecture that addresses indirect prompt injection and unauthorized transactions.

3. After a model update, fraud detection improves overall but produces a sharp increase in false accusations against a small customer group. Propose an incident-response, governance, and remediation plan.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes robustness from reliability?

A. Robustness concerns resistance to perturbations, whereas reliability concerns consistent intended performance under specified conditions.

B. Robustness concerns legal compliance, whereas reliability concerns privacy.

C. Robustness means the system never fails, whereas reliability concerns only security.

D. Robustness applies only to hardware, whereas reliability applies only to data.

Correct Answer: A. Robustness concerns resistance to perturbations, whereas reliability concerns consistent intended performance under specified conditions.

Question 2

A malicious contributor inserts specially labelled examples into a training dataset so that a classifier misidentifies a particular trigger pattern. What is this primarily an example of?

A. Model calibration

B. Data poisoning and possible backdoor creation

C. Model extraction

D. Concept drift

Correct Answer: B. Data poisoning and possible backdoor creation

Question 3

Why is treating retrieved web content as system instruction dangerous in a tool-using language model?

A. Retrieved content always reduces model accuracy.

B. It allows untrusted content to influence actions that may have privileged effects.

C. It prevents the model from generating natural language.

D. It guarantees that confidential data will be deleted.

Correct Answer: B. It allows untrusted content to influence actions that may have privileged effects.

Question 4

Which control most directly limits the damage if an AI assistant is manipulated into requesting an unauthorized payment?

A. Increasing the model’s parameter count

B. Using a larger retrieval database

C. Independent authorization checks, transaction limits, and confirmation outside the model

D. Removing all system logs

Correct Answer: C. Independent authorization checks, transaction limits, and confirmation outside the model

Question 5

An input differs substantially from the data used to train a predictive model. This condition is best described as:

A. Out-of-distribution behaviour

B. Model watermarking

C. Dependency pinning

D. White-box extraction

Correct Answer: A. Out-of-distribution behaviour

Question 6

Which statement about human oversight is most accurate?

A. Any approval button constitutes meaningful oversight.

B. Human oversight is effective only when reviewers have authority, expertise, time, and relevant information.

C. Human oversight eliminates the need for technical safeguards.

D. Human oversight is unnecessary for high-impact decisions if accuracy is high.

Correct Answer: B. Human oversight is effective only when reviewers have authority, expertise, time, and relevant information.

Question 7

What is the main purpose of defence in depth?

A. To make a model larger and more expensive to copy

B. To ensure that failure of one control does not expose the whole system

C. To replace testing with documentation

D. To guarantee perfect fairness across all populations

Correct Answer: B. To ensure that failure of one control does not expose the whole system

Question 8

Which situation most clearly illustrates systemic AI risk?

A. One user mistypes a question to a chatbot.

B. Several institutions rely on the same flawed model and simultaneously make harmful decisions during a crisis.

C. A developer changes a variable name in a test script.

D. A model takes slightly longer to respond to one request.

Correct Answer: B. Several institutions rely on the same flawed model and simultaneously make harmful decisions during a crisis.

Question 9

What is the safest interpretation of an abstention mechanism?

A. It proves that the model understands the input.

B. It allows the system to decline or escalate when prediction conditions are inadequate.

C. It removes the need for monitoring.

D. It guarantees that adversarial inputs are detected.

Correct Answer: B. It allows the system to decline or escalate when prediction conditions are inadequate.

Question 10

Which practice best supports trustworthy incident response?

A. Delete logs immediately to protect the organization’s reputation.

B. Blame the operator without examining system design.

C. Contain the issue, preserve appropriate evidence, communicate responsibly, and analyse underlying causes.

D. Continue deployment until a perfect replacement model is available.

Correct Answer: C. Contain the issue, preserve appropriate evidence, communicate responsibly, and analyse underlying causes.