AI Security, Analytics and Emerging Developments

Introduction

Artificial intelligence (AI) is changing both sides of cybersecurity. Security teams use machine learning to identify anomalies, prioritise vulnerabilities, correlate events and support testing. At the same time, attackers use AI to generate convincing phishing messages, discover weaknesses, automate reconnaissance and manipulate AI-enabled applications.

This creates a dual responsibility. Security professionals must use analytics and AI effectively, but also understand their limitations, govern their use and protect AI systems as valuable application components. An AI security programme therefore includes two related domains:

1. Security for AI: protecting data, models, prompts, pipelines, interfaces and infrastructure.
2. AI for security: applying AI and analytics to detection, testing, response and risk prioritisation.

The central principle is that AI should augment accountable human decision-making, not replace security governance, evidence-based validation or professional judgement.

Learning Objectives

By the end of this chapter, students should be able to:

• distinguish rule-based analytics, statistical detection and machine learning;
• evaluate AI outputs using operationally meaningful metrics;
• apply AI responsibly to detection, testing and vulnerability prioritisation;
• explain prompt injection, unsafe output, data leakage and model-supply-chain threats;
• design controls for AI-enabled web applications;
• assess passkeys, confidential computing and post-quantum readiness;
• analyse the opportunities and limitations of automated security operations; and
• make defensible technical, ethical and governance decisions concerning AI security.

1. Security Analytics as an Evidence Discipline

Security analytics is the systematic collection, transformation and interpretation of security-relevant data. Typical sources include authentication logs, HTTP requests, endpoint telemetry, cloud control-plane events, vulnerability scanners, source-code repositories and identity-provider records.

A useful analytics pipeline has six stages:

1. Collection – acquire events with reliable timestamps, identities and source context.
2. Normalisation – convert different formats into a consistent schema.
3. Enrichment – add asset criticality, threat intelligence, ownership and geographical or business context.
4. Detection – identify rules, deviations, sequences or relationships that may indicate risk.
5. Investigation – test hypotheses against additional evidence.
6. Action and feedback – contain, remediate, close or escalate the event, then use the result to improve detection.

Analytics is not synonymous with AI. A rule such as “deny a request containing a known malicious signature” is deterministic analytics. Statistical analytics may identify an unusual login time or volume. Machine learning learns patterns from data, while generative AI produces text, code or other content. These approaches can be combined, but each has different failure modes.

Detection quality

For a binary detection task, a true positive is a correctly identified malicious event, a false positive is a benign event incorrectly flagged, a true negative is a correctly ignored benign event and a false negative is a missed malicious event.

Precision measures how many alerts are useful; recall measures how much of the relevant malicious activity is detected. These measures assume labelled data and a clearly defined positive class. They do not by themselves measure investigation cost, alert delay or business impact.

The harmonic mean of precision and recall is the F1 score:

The F1 score is useful when both precision and recall matter and when they should receive equal weight. It can be misleading when false negatives are substantially more costly than false positives, or when the positive class is extremely rare. In operational security, analysts should also examine alert volume, mean time to acknowledge, mean time to contain, calibration, coverage and the cost of errors.

For example, if a detector produces TP = 80, FP = 20 and FN = 40:

• Precision = 80 ÷ 100 = 0.80, or 80%.
• Recall = 80 ÷ 120 ≈ 0.667, or 66.7%.
• F1 ≈ 2 × (0.80 × 0.667) ÷ (0.80 + 0.667) ≈ 0.727.

This detector is reasonably precise but misses approximately one-third of the labelled malicious events. A deployment decision requires knowledge of the threat, staffing and consequences of missed activity.

Baselines and anomalies

Anomaly detection compares an observation with an expected baseline. A baseline may represent normal requests per minute, usual administrative actions or typical data-transfer volumes. Baselines must account for seasonality, maintenance windows, deployments, holidays and legitimate organisational change. A model trained only on ordinary working hours may generate excessive alerts during a planned release.

A security anomaly is not automatically an attack. It is a deviation requiring investigation. This distinction is essential because attackers can imitate normal behaviour, while legitimate users can behave unusually.

2. AI-Assisted Detection, Testing and Prioritisation

Detection

AI can assist with:

• clustering similar alerts;
• extracting entities from logs;
• identifying suspicious sequences across systems;
• summarising incidents for analysts;
• detecting unusual authentication or API behaviour; and
• recommending investigative queries.

A safe design separates evidence from interpretation. Raw events should be retained, cryptographically protected where appropriate, and linked to the AI-generated explanation. Analysts must be able to inspect why an alert was generated and reproduce the relevant query or model version.

Generative AI should not be treated as an authoritative source of facts. It may hallucinate events, misinterpret log syntax or omit important evidence. Retrieval from approved internal data, constrained tools, output validation and human review reduce these risks.

Security testing

AI can accelerate secure coding and testing by generating test cases, explaining code paths, identifying suspicious data flows and proposing abuse cases. It may support static analysis, dynamic testing and fuzzing, but generated tests require review. A model can produce syntactically valid tests that lack meaningful coverage or incorrectly classify a vulnerability.

A professional workflow is:

1. define the security property or abuse case;
2. use AI to propose tests or hypotheses;
3. execute tests in an isolated, authorised environment;
4. verify findings with independent tools or manual analysis;
5. record evidence, affected versions and reproduction steps; and
6. retest after remediation.

AI must never be used to conduct unauthorised scanning or exploitation. Scope, approval, rate limits, data handling and emergency contacts should be defined before testing.

Vulnerability prioritisation

Severity scores are useful but incomplete. A vulnerability in a critical internet-facing payment service may deserve earlier action than a higher-scored issue in an isolated laboratory system. Prioritisation should combine technical severity with exposure, exploit evidence, asset importance, reachability, compensating controls and remediation cost.

A transparent risk model can use a weighted score:

Here, R is a prioritisation score; S is technical severity; E is exploit evidence; A is asset criticality; X is exposure or reachability; C is the strength of compensating controls; and w₁ through w₅ are organisation-approved weights. The variables must be normalised to a documented scale, such as 0 to 1, before combining them. This is a decision-support model, not a universal risk law. Weights should be validated against incidents and reviewed when the threat environment changes.

3. Threats to AI-Enabled Applications

An AI-enabled application usually contains a user interface, application logic, model or model API, prompts, retrieval components, tools, data stores, monitoring and an infrastructure or supply chain. Each component introduces attack surfaces.

Prompt injection

Prompt injection occurs when untrusted content influences an AI system to disregard intended instructions or perform an unauthorised action. Direct injection comes from a user prompt. Indirect injection is embedded in retrieved web pages, documents, emails or database content.

The fundamental design error is treating instructions and data as if they had equal authority. A retrieved document should be treated as data, not as a trusted command. Controls include:

• separating system instructions, developer instructions, user content and retrieved content;
• using allow-listed tools with narrowly defined parameters;
• requiring authorisation outside the model before consequential actions;
• limiting access according to the user’s identity and permissions;
• validating tool arguments and outputs;
• applying confirmation for external communication, deletion, payments or code deployment; and
• testing direct and indirect injection scenarios.

No prompt wording can guarantee security. Prompt instructions are one layer in a defence-in-depth design, not an access-control mechanism.

Unsafe output and excessive agency

Models can generate malicious code, discriminatory recommendations, fabricated citations, dangerous instructions or content that violates policy. Output should therefore be validated according to context. For example, generated SQL should not be executed directly; generated HTML must be safely encoded; and generated access decisions require deterministic policy enforcement.

An agent with excessive permissions can convert a language error into a serious incident. The principle of least privilege applies to tools, tokens, network destinations, file access and transaction limits. Agents should operate in isolated environments and produce auditable action records.

Sensitive-information disclosure

Data leakage may occur through prompts, logs, training data, retrieval indexes, model outputs, debug traces or third-party APIs. Controls include data minimisation, classification, redaction, tenant isolation, encryption, retention limits, access controls and contractual review of providers. Secrets should be detected before prompt submission and should never be placed in system prompts merely because the model is expected to keep them confidential.

Organisations should determine whether provider data is retained, used for training, transferred internationally or accessible to support personnel. Privacy impact assessments may be required where personal data is processed.

Model attacks

Important model threats include:

• Data poisoning: malicious or low-quality training data alters model behaviour.
• Backdoors: a trigger causes hidden undesirable behaviour.
• Evasion: carefully crafted inputs avoid detection.
• Model extraction: repeated queries approximate a proprietary model.
• Membership inference: an attacker attempts to determine whether a record was used in training.
• Model inversion: outputs are used to infer characteristics of training data.
• Denial of service: expensive or excessive inputs consume resources.

Risk reduction includes provenance checks, dataset validation, access controls, rate limiting, monitoring, adversarial testing, privacy-preserving techniques where appropriate, model signing and controlled release processes.

AI supply-chain security

The supply chain includes datasets, pretrained models, model weights, packages, containers, plugins, evaluation sets, infrastructure and external APIs. A compromised dependency may introduce malicious code or a model backdoor.

A defensible supply-chain process maintains an inventory, pins and verifies dependencies, scans packages and images, records provenance, separates build and production credentials, signs artefacts, reviews licences and monitors for withdrawal or compromise. Model cards, dataset documentation and software bills of materials support transparency but do not prove that an artefact is safe.

4. A Secure Architecture for AI Applications

A secure AI application should use layered controls:

• Identity layer: authenticate users and services; enforce tenant and role boundaries.
• Policy layer: make deterministic decisions about authorisation, data access and permitted actions.
• Orchestration layer: control prompts, retrieval, tool calls, budgets and workflows.
• Model layer: select, isolate, evaluate and monitor models.
• Data layer: classify, minimise, encrypt and govern information.
• Execution layer: sandbox code and tools; restrict network and filesystem access.
• Monitoring layer: record prompts and actions subject to privacy rules, detect abuse and support incident response.

The model should not directly decide whether it is allowed to access a record. The application should enforce that decision before retrieval. Similarly, the model should not directly issue a payment or modify production code; a policy-controlled service should validate and approve the request.

Evaluation should include normal functionality, security abuse cases, privacy tests, reliability, fairness where relevant and resilience under changing inputs. Test sets should include unseen examples and adversarial examples, while avoiding the publication of sensitive test data.

5. Emerging Developments

Passkeys

Passkeys use public-key credentials, commonly through FIDO2/WebAuthn mechanisms. The private key remains under control of the authenticator, while the service stores a public key. Authentication is bound to the legitimate relying-party domain, reducing phishing and password-reuse risk.

Advantages include resistance to credential phishing, elimination of server-side password databases and convenient device-based authentication. Limitations include account recovery, device loss, synchronisation choices, accessibility, legacy integration and the need for careful enrolment and account-recovery procedures. Passkeys should be combined with strong account lifecycle management and monitoring for suspicious enrolment.

Confidential computing

Confidential computing uses hardware-supported trusted execution environments to protect data while it is being processed. Encryption at rest protects stored data and encryption in transit protects network communication; confidential computing aims to reduce exposure during computation.

Attestation can allow a service to verify that an approved workload is running before releasing secrets. However, trusted execution environments do not automatically secure an application. Vulnerabilities in the workload, incorrect attestation policy, side channels, key-management errors and compromised administrators may remain relevant. Confidential computing is particularly useful for multi-party analytics, sensitive AI inference and regulated workloads when its threat model matches the use case.

Post-quantum readiness

Large-scale quantum computers could threaten widely used public-key algorithms based on integer factorisation or discrete logarithms. Symmetric cryptography is affected differently; sufficiently large keys and appropriate designs provide greater resilience. Migration is not an instantaneous algorithm swap because certificates, protocols, libraries, hardware and embedded devices may have long lifetimes.

A post-quantum readiness programme should:

1. inventory cryptographic algorithms and where they are used;
2. identify sensitive data requiring long-term confidentiality;
3. assess vendors, certificates, protocols and constrained devices;
4. prefer cryptographic agility in new designs;
5. test standardised post-quantum algorithms and hybrid approaches where appropriate; and
6. create a staged migration and validation plan.

“Harvest now, decrypt later” makes long-lived confidential data relevant before a cryptographically capable quantum computer exists. Organisations should follow current standards and authoritative migration guidance rather than adopting unreviewed experimental algorithms.

Security automation and autonomous operations

Automation can enrich alerts, rotate credentials, quarantine hosts, open tickets and apply known remediations. Fully autonomous action is appropriate only where the action is reversible, well scoped, observable and low risk. High-impact actions should require approval or strong policy gates.

Automation creates its own risks: cascading errors, corrupted feedback loops, hidden bias, overconfident recommendations and attacks against orchestration systems. Every automated action should have an owner, a rollback path, an audit trail, rate limits and a tested failure mode.

6. Governance, Ethics and Regulation

Responsible AI security requires accountability, transparency, privacy and proportionality. Organisations should document the purpose of each model, data sources, intended users, known limitations, evaluation results, approval status and retirement conditions.

Governance should address:

• lawful and fair processing of personal data;
• retention and access to prompts and outputs;
• human oversight for consequential decisions;
• explainability appropriate to the risk;
• discrimination and unequal error rates;
• intellectual-property and licensing obligations;
• incident reporting and supplier accountability; and
• independent review of high-risk systems.

AI-generated evidence should be marked as such and independently verified. Staff must understand that confidential information entered into an unapproved service may become a data-protection incident. Policies should define approved tools, prohibited data, review requirements and escalation routes.

Applied Case: Illustrative Retail-Banking Support Agent

Background

A bank deploys an AI assistant that answers product questions, retrieves account-specific information and submits service requests. It uses a language model, a retrieval index, customer records and tools for address changes and card replacement.

Problem

During testing, a document in the retrieval index contains hidden instructions telling the assistant to disclose customer records and email them to an external address. In another test, a user persuades the assistant to submit an address change without adequate verification. Logs also contain full account numbers, increasing the impact of a possible breach.

Analysis

The first issue is indirect prompt injection. The second is excessive agency and a failure to separate conversational assistance from transaction authorisation. The third is data minimisation and logging failure. The model is not the correct component to enforce identity, permission or transaction policy.

Approach and solution

The bank removes instruction-like content from trusted retrieval sources or labels retrieved material explicitly as untrusted data. Retrieval is filtered by customer identity and authorisation before content reaches the model. Address changes require a separate authenticated workflow, deterministic validation and customer confirmation. The assistant can explain the process but cannot complete it directly.

Tool calls are allow-listed, parameter-validated and rate-limited. Production tools run through a policy gateway. Account numbers are tokenised or masked in logs, with restricted access and defined retention. Red-team tests cover direct injection, indirect injection, data-exfiltration attempts, privilege escalation and denial of service. A monitoring system links model output, retrieved sources, tool calls and final policy decisions.

Outcome

The assistant continues to answer low-risk questions, while high-impact operations are transferred to controlled workflows. Testing identifies more attempted injections, but the attempts no longer produce unauthorised disclosure or account changes. The bank retains evidence for investigation and periodically reassesses the model, retrieval content and supplier controls.

Learning points

• Prompt instructions are not a substitute for authorisation.
• Retrieved data must be treated as potentially hostile.
• High-impact actions require deterministic policy controls.
• Logging must support investigation without unnecessarily exposing sensitive data.
• AI security is a system-engineering and governance problem, not merely a model-selection problem.

Practical Best Practices

1. Maintain an inventory of AI models, datasets, prompts, tools, owners and suppliers.
2. Classify data before it enters prompts, retrieval indexes or logs.
3. Apply least privilege to users, services, agents and tools.
4. Keep policy enforcement outside the language model.
5. Validate all model-generated code, queries, commands and decisions.
6. Test direct and indirect prompt injection continuously.
7. Preserve raw evidence and model/version metadata for investigations.
8. Measure precision, recall, coverage, analyst workload and business impact.
9. Use staged deployment, canary releases and rollback for model changes.
10. Review passkey recovery, confidential-computing assumptions and cryptographic migration plans.
11. Require human approval for irreversible or high-impact actions.
12. Train staff to recognise hallucination, data leakage and automation bias.

Challenges and Limitations

AI systems are probabilistic, dependent on data quality and vulnerable to distribution shift. Attackers adapt to detectors, labels may be incomplete and rare attacks provide little training data. Accuracy in a laboratory may not survive production conditions. Explainability methods can be incomplete or misleading, and a fluent explanation is not proof of correctness.

Automation can also concentrate risk: a flawed model connected to powerful tools may cause harm faster than a human operator. Conversely, excessive approval requirements may cause alert fatigue and reduce the value of automation. Security architects must therefore balance safety, usability, latency, cost and operational capacity.

Chapter Summary

Security analytics provides the evidence pipeline through which events are collected, normalised, enriched, detected, investigated and acted upon. Precision, recall and F1 help evaluate detectors, but operational decisions must also consider cost, coverage, delay and business impact. AI can improve detection, testing and vulnerability prioritisation when its outputs are validated and linked to evidence.

AI-enabled applications introduce threats including prompt injection, unsafe output, excessive agency, sensitive-information disclosure, data poisoning, model extraction and supply-chain compromise. Secure designs place identity, authorisation and policy enforcement outside the model; isolate tools; validate outputs; minimise data; and maintain auditability.

Passkeys reduce phishing and password risks but require sound recovery and enrolment controls. Confidential computing protects data during processing under a defined hardware and attestation threat model. Post-quantum readiness requires cryptographic inventory, agility and staged migration. Automation should be bounded by least privilege, approval gates, observability and rollback. Responsible AI security combines engineering, governance, privacy, ethics and continuous evaluation.

Key Terms

Security analytics: Systematic analysis of security-relevant data to support detection and decisions.

True positive: A malicious or relevant event correctly identified by a detector.

False positive: A benign event incorrectly identified as malicious.

Precision: The proportion of positive alerts that are true positives.

Recall: The proportion of relevant positive events that are detected.

F1 score: The harmonic mean of precision and recall.

Anomaly detection: Identification of observations that differ from an expected baseline.

Prompt injection: Manipulation of an AI system through instructions in user or retrieved content.

Indirect prompt injection: Injection delivered through content retrieved from an external or untrusted source.

Excessive agency: Giving an AI component more authority or capability than necessary.

Data poisoning: Deliberate corruption of training or evaluation data.

Model extraction: Attempt to reproduce a model through repeated queries.

Supply-chain attack: Compromise introduced through a dependency, provider, model, package or build component.

Passkey: A public-key authentication credential designed to resist phishing.

Confidential computing: Protection of data during processing using trusted execution environments.

Attestation: Evidence that a specified software or hardware environment is running.

Post-quantum cryptography: Cryptography designed to resist attacks from relevant quantum algorithms.

Cryptographic agility: The ability to replace cryptographic algorithms with limited redesign.

Human-in-the-loop: Human review or approval incorporated into an automated process.

Model drift: Decline in model performance as the environment or data distribution changes.

Review Questions

Short Answer Questions

1. Distinguish precision from recall in security detection.
2. Why is prompt wording alone insufficient to secure an AI agent?
3. State two controls that reduce sensitive-information disclosure.
4. What is the purpose of cryptographic agility?
5. Why should retrieved documents be treated as untrusted data?

Long Answer Questions

1. Explain the stages of a security analytics pipeline and discuss the role of feedback.
2. Compare rule-based detection, anomaly detection and machine-learning detection.
3. Describe a secure architecture for an AI-enabled web application.
4. Evaluate the benefits and limitations of passkeys and confidential computing.
5. Develop a post-quantum readiness plan for an organisation with long-lived sensitive data.

Analytical / Application Questions

1. A detector records TP = 90, FP = 30 and FN = 10. Calculate its precision, recall and F1 score, and explain whether these measures are sufficient for deployment.
2. Design controls for an AI customer-service agent that can retrieve records and initiate refunds. Identify which actions require deterministic policy enforcement.
3. An AI vulnerability-prioritisation system ranks an isolated high-severity flaw above a medium-severity flaw in an internet-facing payment service. Analyse why the ranking may be unsafe and propose a better decision process.

Multiple Choice Questions with Answers

Question 1

Which measure answers the question: “Of the alerts classified as positive, what proportion were correct?”

A. Recall

B. Precision

C. Availability

D. Latency

Correct Answer: B. Precision

Question 2

Which control most directly prevents a language model from authorising an unauthorised account change?

A. A longer system prompt

B. A larger language model

C. A deterministic policy gateway outside the model

D. Disabling audit logs

Correct Answer: C. A deterministic policy gateway outside the model

Question 3

An instruction hidden in a retrieved web page that attempts to control an AI assistant is an example of what threat?

A. Indirect prompt injection

B. Password spraying

C. Model compression

D. Certificate pinning

Correct Answer: A. Indirect prompt injection

Question 4

Which practice best reduces the risk of unsafe model-generated SQL being executed?

A. Execute the SQL without inspection

B. Use output validation and parameterised database access

C. Give the model administrator privileges

D. Remove all database logs

Correct Answer: B. Use output validation and parameterised database access

Question 5

Which statement about passkeys is correct?

A. They require the service to store the user’s password

B. They are based on public-key authentication and are designed to resist phishing

C. They eliminate the need for account recovery planning

D. They work by sending a reusable secret by email

Correct Answer: B. They are based on public-key authentication and are designed to resist phishing

Question 6

Confidential computing primarily aims to protect data during which state?

A. Processing

B. Printing

C. Archiving paper records

D. Physical disposal

Correct Answer: A. Processing

Question 7

What is the purpose of a post-quantum readiness inventory?

A. To identify where cryptographic algorithms and dependencies are used

B. To remove all encryption immediately

C. To replace every password with a PIN

D. To guarantee that quantum computers cannot be built

Correct Answer: A. To identify where cryptographic algorithms and dependencies are used

Question 8

Which is an example of model extraction?

A. Encrypting a model at rest

B. Attempting to approximate a proprietary model through repeated queries

C. Deleting old audit logs

D. Adding a new firewall rule

Correct Answer: B. Attempting to approximate a proprietary model through repeated queries

Question 9

A detector has TP = 80, FP = 20 and FN = 40. What is its precision?

A. 40%

B. 66.7%

C. 80%

D. 90%

Correct Answer: C. 80%

Question 10

Which practice is most appropriate for a high-impact automated remediation?

A. Give the agent unrestricted permissions

B. Use approval gates, least privilege, auditability and rollback

C. Disable monitoring to reduce latency

D. Allow irreversible actions without confirmation

Correct Answer: B. Use approval gates, least privilege, auditability and rollback