AI, Automation and Emerging Security Developments

Introduction

Cybersecurity is increasingly shaped by systems that can analyse data, recommend actions and execute controls at machine speed. Artificial intelligence (AI), security orchestration, automation and response (SOAR), extended detection and response (XDR), secure access service edge (SASE), passwordless authentication and post-quantum cryptography are changing how organisations design and operate security. At the same time, adversaries use AI to generate convincing phishing messages, discover vulnerabilities, evade detection and create deepfakes. AI systems themselves introduce risks involving poisoned data, manipulated inputs, model theft, privacy loss and unsafe automation.

The central professional challenge is therefore not simply to “adopt AI”. It is to establish trustworthy human–machine collaboration. Security teams must understand what automation can do reliably, where human judgement remains essential, how evidence is validated, and how new technologies affect governance, resilience and workforce capability.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain AI-assisted detection, automated response and security orchestration.
2. Evaluate the operational benefits and limitations of machine-learning security systems.
3. Analyse adversarial AI, deepfakes and model-security threats.
4. Compare SOAR, XDR and SASE as complementary rather than interchangeable approaches.
5. Explain quantum threats and the rationale for post-quantum migration.
6. Assess passwordless authentication architectures and implementation risks.
7. Recommend governance, assurance and workforce practices for emerging security technologies.

1. AI-Assisted Detection

1.1 From signatures to behavioural analysis

Traditional detection relies heavily on signatures: known byte sequences, malicious domains, file hashes or explicit rules. Signature methods are fast and interpretable, but they are weak against novel malware, polymorphism and attacks using legitimate administrative tools.

AI-assisted detection expands the evidence base. Systems may analyse:

• endpoint processes and parent–child relationships;
• authentication sequences and access times;
• network flows, DNS queries and application behaviour;
• cloud configuration and API activity;
• email language, sender reputation and attachment behaviour; and
• relationships among users, devices, identities and resources.

Machine learning can be supervised, unsupervised or semi-supervised. Supervised models learn from labelled examples such as “malicious” and “benign”. Unsupervised methods identify clusters or anomalies without requiring complete labels. Semi-supervised approaches combine a smaller labelled set with larger unlabelled data.

A useful conceptual model is risk scoring. A detection system combines multiple signals into an estimate of the likelihood or priority of an incident. The score is not automatically a probability unless it has been statistically calibrated. This distinction is important: a score of 0.8 may mean “high priority” rather than “an 80% chance of compromise”.

1.2 Detection quality and operational metrics

Security detection is affected by class imbalance: genuine attacks are usually much rarer than normal events. Accuracy can therefore be misleading. If 99.9% of events are benign, a system that labels everything benign has 99.9% accuracy but detects nothing.

For a binary classifier:

Here, TP is true positives, FP is false positives, and FN is false negatives. Precision measures how many alerts classified as malicious are actually malicious. Recall measures how many malicious events are detected. The F1 score is the harmonic mean of precision and recall:

These metrics assume clearly defined labels and a fixed decision threshold. In operations, the threshold should reflect consequences. A low threshold may improve recall but overwhelm analysts with false alerts. A high threshold may reduce workload but allow attacks to remain undetected. Organisations should also measure mean time to detect (MTTD), mean time to respond (MTTR), alert closure quality and analyst workload.

1.3 Explainability and analyst trust

A model that produces an alert without supporting evidence is difficult to investigate. Useful explanations may include the features that influenced a decision, the timeline of activity, comparable historical events and the confidence or uncertainty of the model. Explainability does not prove that a model is correct; it helps an analyst challenge or validate the output.

Security data also creates privacy concerns. User behaviour analytics can reveal working patterns, health-related inferences or union activity. Collection must therefore be necessary, proportionate, access-controlled and governed by applicable privacy law and organisational policy.

2. Automated Response and Security Orchestration

2.1 SOAR and playbooks

Security orchestration coordinates tools, while automation executes repeatable actions. A SOAR platform may receive an alert from an endpoint system, enrich it with threat-intelligence data, check identity and asset information, create a case, and—if conditions are satisfied—disable an account or isolate a device.

A playbook is a documented sequence of actions, decisions and approvals. Strong playbooks specify:

1. the trigger and required evidence;
2. enrichment sources and validation steps;
3. decision thresholds;
4. permitted automated actions;
5. approval or human-review points;
6. rollback procedures;
7. evidence and audit-log requirements; and
8. post-incident improvement activities.

Automation should be graduated. Low-risk actions, such as adding context to an alert or blocking a confirmed malicious domain, can often be automated. High-impact actions, such as disabling a hospital identity or shutting down a production server, normally require human approval or multiple independent signals.

2.2 The danger of unsafe automation

Automation can amplify mistakes. A faulty rule may quarantine thousands of legitimate endpoints, delete evidence, lock out privileged users or interrupt safety-critical services. Common failure modes include stale threat intelligence, incorrect asset classification, excessive permissions, circular playbooks and actions that cannot be reversed.

A secure automation design applies least privilege, rate limits, simulation or “dry-run” modes, change control, segregation of duties and immutable logging. Every destructive action should have a defined scope and recovery path. Human approval is not a complete safeguard if reviewers cannot understand the evidence or are pressured to approve automatically generated recommendations.

2.3 Security orchestration as socio-technical engineering

SOAR does not replace security operations. It changes the distribution of work. Analysts spend less time copying information between consoles and more time validating hypotheses, improving detections and handling exceptional cases. The organisation must redesign processes, training and performance measures rather than merely install a platform.

3. XDR and SASE

3.1 Extended Detection and Response

XDR correlates telemetry across several security domains, commonly endpoint, email, identity, network and cloud services. Its value arises from cross-domain context. An isolated suspicious login may be ambiguous; the same login followed by mailbox-rule creation, unusual endpoint execution and data transfer is more significant.

XDR may be vendor-integrated or open, using data connectors and a common analysis layer. Evaluation criteria include telemetry coverage, data retention, correlation quality, interoperability, investigation workflow, response controls and the ability to export evidence. XDR is not synonymous with a single product, and a broad marketing label does not guarantee complete visibility.

3.2 Secure Access Service Edge

SASE is an architectural approach that combines networking and security capabilities delivered close to users, devices and applications. Typical capabilities include software-defined wide-area networking, secure web gateways, cloud access security brokerage, zero-trust network access and firewall-as-a-service.

SASE responds to distributed work and cloud adoption. Instead of assuming that traffic should pass through a central corporate perimeter, it evaluates identity, device posture, application, location, risk and policy. It does not eliminate the need for endpoint security, identity governance or secure application design.

3.3 Distinguishing the approaches

These approaches can be integrated. SASE may provide telemetry and policy enforcement; XDR may correlate events; SOAR may coordinate response. They remain conceptually distinct and should be assessed against defined business and security outcomes.

4. Adversarial AI and Model-Security Risks

4.1 Attacks against AI systems

Adversarial AI refers both to the malicious use of AI and to attacks against AI-enabled systems. Important categories include:

• Data poisoning: training data is altered so that the model learns incorrect or attacker-favourable behaviour.
• Evasion: carefully modified inputs cause an operational model to misclassify an event.
• Prompt injection: untrusted content attempts to manipulate a language model into ignoring instructions or revealing information.
• Model extraction: repeated queries are used to approximate a proprietary model.
• Membership inference: an attacker attempts to determine whether particular data appeared in training.
• Sensitive-data leakage: prompts, logs or outputs reveal confidential information.
• Supply-chain compromise: a model, dataset, plugin or dependency contains malicious code or hidden behaviour.
• Automation abuse: attackers exploit an AI agent’s tools, credentials or excessive permissions.

Threat modelling must cover the complete AI lifecycle: data collection, labelling, training, deployment, monitoring, updating and retirement.

4.2 Generative AI and deepfakes

Generative models can produce realistic text, audio, images and video. Attackers may impersonate executives, create fraudulent payment instructions, automate reconnaissance or generate tailored social-engineering campaigns. Deepfakes are especially dangerous where a person’s voice or appearance is treated as proof of identity.

Defences should not rely solely on detecting synthetic media. Detection tools have imperfect generalisation and may fail when generation methods change. Stronger controls include independent out-of-band verification, transaction limits, dual approval, hardware-backed identity, provenance metadata where available and staff training that emphasises process rather than appearance.

4.3 Securing machine-learning pipelines

A defensible AI pipeline should include:

• authenticated and integrity-checked data sources;
• data provenance and documented labelling decisions;
• separation of training, validation and test data;
• access control for models, prompts and datasets;
• adversarial and robustness testing;
• monitoring for drift and abnormal inputs;
• versioning and reproducible deployment;
• red-team exercises and incident-response procedures; and
• a documented process for disabling or rolling back a model.

Model performance can degrade when the operating environment changes. This is called drift. A model trained on one workforce, geography or software estate may perform poorly after a major organisational change. Monitoring should therefore assess both prediction quality and changes in the underlying data distribution.

5. Quantum Threats and Post-Quantum Security

5.1 Why quantum computing matters

Large-scale fault-tolerant quantum computers could threaten widely used public-key algorithms. Shor’s algorithm provides a theoretical route to solving integer factorisation and discrete logarithm problems more efficiently than known classical methods. This affects RSA, finite-field Diffie–Hellman and elliptic-curve cryptography.

Symmetric cryptography is affected differently. Grover’s algorithm offers a quadratic search speed-up in idealised conditions. A practical response is generally to use sufficiently large symmetric key sizes; the impact is not equivalent to the complete break of public-key systems.

The “harvest now, decrypt later” threat is important: adversaries may collect encrypted traffic today and attempt decryption when capable quantum systems become available. Long-lived confidential information therefore requires migration planning before a cryptographically relevant quantum computer exists.

5.2 Migration principles

Post-quantum cryptography (PQC) uses classical algorithms believed to resist known quantum attacks. Migration requires cryptographic inventory, identification of long-lived data, supplier assessment, protocol upgrades, certificate planning and algorithm agility. Hybrid modes may combine a classical exchange with a post-quantum exchange during transition, but hybrid security depends on correct protocol design and implementation.

Quantum risk management should not assume a precise arrival date. Organisations should prioritise systems according to confidentiality lifetime, replacement difficulty, exposure and supply-chain dependency. Cryptographic agility—the ability to replace algorithms without redesigning every application—is a strategic security property.

6. Passwordless Authentication

Passwords are vulnerable to reuse, phishing, credential stuffing, guessing and support-driven resets. Passwordless authentication uses alternatives such as passkeys based on public-key cryptography, hardware security keys, smart cards or device-bound authenticators.

In a typical public-key design, the authenticator retains a private key and the service stores a public key. During authentication, the service sends a challenge; the authenticator signs it after local user verification, such as a biometric or device PIN. The biometric generally unlocks the local authenticator rather than being transmitted to the service.

Benefits include resistance to ordinary phishing, reduced password-reuse risk and stronger binding between an authenticator and a service origin. Risks remain: account-recovery processes may reintroduce weak links; device loss can create availability problems; poorly governed administrator enrolment can enable takeover; and users may be deceived into approving fraudulent requests. Deployment should include recovery controls, device lifecycle management, phishing-resistant administrator access, accessibility support and tested break-glass procedures.

7. Governance, Ethics and Regulation

AI-enabled security must be governed as a high-impact socio-technical capability. Core governance questions include:

• What decision is being automated?
• Who is accountable for errors?
• What evidence supports the decision?
• Can an affected person challenge it?
• What data is collected and for how long?
• Is the system proportionate to the threat?
• What happens when the model is unavailable or wrong?

Organisations should maintain an AI system register, risk classification, data-protection assessment, model documentation, supplier assurance, access reviews and incident procedures. Security controls should align with recognised risk-management practices, applicable privacy and cybercrime law, sector regulation, contractual requirements and emerging AI regulation.

Ethical concerns include discriminatory detection, excessive employee surveillance, opaque automated decisions and unequal access to secure authentication. Accuracy alone is not enough: a system may be statistically effective but unacceptable if it imposes disproportionate harm on a protected group or denies legitimate access without appeal.

8. Workforce and Professional Practice

The emerging security workforce requires both technical and judgement-based capabilities. Important skills include:

• identity, cryptography, cloud and network architecture;
• data engineering and machine-learning evaluation;
• detection engineering and incident response;
• scripting, APIs and secure automation;
• privacy, governance and risk communication;
• adversarial testing and threat modelling; and
• forensic reasoning and evidence handling.

Security analysts increasingly act as investigators and automation designers. They must understand uncertainty, question model outputs and communicate risk to non-specialists. Managers need to evaluate total operating cost, integration effort, dependency concentration, staffing and resilience—not merely product features.

Applied Case: Illustrative Regional Health Network

Background

A regional health network operates hospitals, clinics, cloud applications and thousands of connected medical and administrative devices. It experiences credential-phishing incidents and receives more than 20,000 security alerts per day.

Problem

The existing team cannot investigate every alert. A proposal recommends fully automated isolation of any endpoint receiving a high machine-learning risk score. Clinical leaders oppose the proposal because an incorrect isolation could interrupt patient care.

Analysis

The security team discovers that the model performs well on historical office endpoints but has limited training data for medical devices. Its score is not calibrated as a probability, and the asset inventory does not reliably distinguish life-critical devices from ordinary workstations. The proposed action therefore combines model uncertainty with high operational impact.

Approach

The network adopts a staged design:

1. AI scores are used for prioritisation, not unconditional isolation.
2. Alerts are enriched with asset criticality, identity and recent activity.
3. Low-risk actions, such as collecting additional telemetry, are automated.
4. Isolation of ordinary endpoints requires two independent signals.
5. Isolation of clinical devices requires human approval and clinical coordination.
6. Every action is logged, reversible and tested in a simulation environment.
7. The team measures precision, recall, analyst workload, MTTD, MTTR and unsafe-action rate.
8. Passkeys are introduced for administrators, with recovery and emergency-access controls.

Outcome

The organisation reduces repetitive investigation and improves response speed without treating the model as an unquestionable authority. It also identifies a data-quality problem and creates a programme to classify assets and improve model coverage.

Learning Points

The case illustrates that successful security automation depends on context, asset criticality, reversibility and governance. A technically impressive model can be unsafe when connected to poorly controlled actions. Human oversight is most valuable at high-impact decision points, not necessarily at every low-risk enrichment step.

Challenges and Limitations

Emerging security technologies face persistent limitations:

• models may produce false positives, false negatives or unstable explanations;
• attackers adapt to detection and poisoning controls;
• automation creates concentration and outage risks;
• cloud and platform dependence can reduce portability;
• telemetry collection may conflict with privacy principles;
• skilled personnel remain scarce;
• vendors may make incompatible claims about XDR, AI or SASE;
• quantum migration is complex and benefits may be difficult to measure; and
• passwordless systems can still fail through recovery, social engineering or compromised endpoints.

A mature organisation treats these technologies as risk-management components rather than replacements for secure architecture, patching, segmentation, identity governance, backup, training and incident response.

Best-Practice Checklist

1. Define the security outcome before selecting a technology.
2. Maintain accurate asset, identity and data inventories.
3. Validate model performance on current, representative data.
4. Distinguish a risk score from a calibrated probability.
5. Use graduated automation based on impact and reversibility.
6. Apply least privilege to playbooks, agents and integrations.
7. Preserve evidence and support rollback.
8. Test models against poisoning, evasion, prompt injection and leakage.
9. Establish independent verification for high-value transactions and deepfake-prone communications.
10. Plan cryptographic inventory and post-quantum migration early.
11. Make recovery a first-class part of passwordless authentication.
12. Measure operational, human and ethical outcomes, not only detection rates.

Chapter Summary

AI-assisted detection improves the ability to identify behavioural patterns across endpoint, identity, network, cloud and application data, but its outputs must be interpreted in context. Precision, recall and F1 score are useful measures, although operational impact, asset criticality and analyst workload are equally important. SOAR coordinates and automates workflows; XDR correlates security telemetry; and SASE provides distributed access and security enforcement. They are complementary approaches rather than interchangeable labels.

Adversarial AI includes poisoning, evasion, prompt injection, model extraction, privacy attacks and abuse of automated tools. Deepfakes increase the need for independent verification and transaction controls. Quantum computing threatens important public-key systems, making cryptographic inventory, agility and post-quantum migration strategic priorities. Passwordless authentication can reduce phishing and reuse risks, but recovery and device governance must be designed carefully.

The responsible adoption of emerging security technology requires human accountability, privacy protection, transparent evidence, tested rollback, supplier assurance and a workforce capable of combining engineering with critical judgement.

Key Terms

Artificial intelligence: Computational systems that perform tasks associated with human reasoning, classification or decision-making.

Adversarial AI: Malicious use of AI or attacks designed to manipulate AI systems.

Automation: Execution of defined actions by software with limited direct human intervention.

Deepfake: Synthetic or manipulated media designed to imitate a real person or event.

Data poisoning: Deliberate corruption of training data to influence model behaviour.

Evasion attack: Manipulation of an input to cause incorrect model classification.

F1 score: Harmonic mean of precision and recall.

False positive: A benign event incorrectly classified as malicious.

Model drift: Decline in model suitability caused by changes in data or operating conditions.

Passwordless authentication: Authentication that does not require a memorised password as the primary credential.

Post-quantum cryptography: Cryptographic algorithms designed to resist known classical and quantum attacks.

Quantum threat: Risk that quantum computation will undermine currently used cryptographic assumptions.

SASE: Architecture combining network and security services near users, devices and applications.

SOAR: Security orchestration, automation and response.

Threat intelligence: Information about threats, adversaries, indicators and attack methods.

XDR: Extended detection and response across multiple security domains.

Prompt injection: Manipulation of model instructions through malicious or untrusted input.

Cryptographic agility: Capability to replace cryptographic algorithms with limited system redesign.

Review Questions

Short Answer Questions

1. Why can accuracy be misleading when evaluating a security classifier?
2. Distinguish SOAR from XDR.
3. What is data poisoning?
4. Why is cryptographic agility important for quantum readiness?
5. State two risks associated with passwordless authentication.

Long Answer Questions

1. Explain how an organisation should design a graduated automated-response programme.
2. Compare SASE, XDR and SOAR, including how they may operate together.
3. Discuss the security and privacy risks of AI-assisted user behaviour analytics.
4. Explain why deepfake detection alone is insufficient for protecting high-value transactions.
5. Describe the main governance requirements for deploying AI in a security operations centre.

Analytical / Application Questions

1. A model produces 90 true positives, 10 false positives and 20 false negatives. Calculate its precision, recall and F1 score, showing the formulas and interpreting the result.
2. Design a response policy for a hospital in which an AI system identifies a high-risk login to a clinical administrator account. Specify automated actions, human approvals, evidence requirements and rollback controls.
3. An organisation has encrypted archives that must remain confidential for 20 years. Develop a prioritised post-quantum migration plan and identify the information needed before selecting replacement algorithms.

Multiple Choice Questions with Answers

Question 1

Which metric measures the proportion of detected positive alerts that are actually positive?

A. Recall

B. Precision

C. Availability

D. Latency

Correct Answer: B. Precision

Question 2

What is the principal purpose of a SOAR playbook?

A. To replace all human analysts

B. To define repeatable investigation and response actions

C. To encrypt every network connection

D. To generate public encryption keys

Correct Answer: B. To define repeatable investigation and response actions

Question 3

Which situation is an example of data poisoning?

A. Changing an input to evade a deployed classifier

B. Stealing a model through repeated queries

C. Injecting misleading records into training data

D. Blocking a security alert from reaching an analyst

Correct Answer: C. Injecting misleading records into training data

Question 4

What is a major characteristic of XDR?

A. It correlates telemetry across security domains

B. It provides only password storage

C. It replaces cryptographic algorithms with quantum hardware

D. It exclusively protects physical buildings

Correct Answer: A. It correlates telemetry across security domains

Question 5

Which practice best reduces the risk of a deepfake payment request?

A. Trusting the executive’s familiar voice

B. Using independent out-of-band verification

C. Disabling all transaction logging

D. Allowing a single employee to approve every payment

Correct Answer: B. Using independent out-of-band verification

Question 6

Which cryptographic systems are most directly threatened by Shor’s algorithm?

A. Public-key systems based on factoring or discrete logarithms

B. All hash functions regardless of output size

C. Physical access cards only

D. Randomly generated passwords only

Correct Answer: A. Public-key systems based on factoring or discrete logarithms

Question 7

In a typical passkey design, what does the service normally store?

A. The user’s biometric image

B. The authenticator’s private key

C. A public key associated with the authenticator

D. The user’s memorised password

Correct Answer: C. A public key associated with the authenticator

Question 8

Why should high-impact automated responses often require human approval?

A. Models never process data quickly enough

B. Incorrect actions can cause serious operational harm

C. Human approval guarantees the model is correct

D. Automation cannot create audit logs

Correct Answer: B. Incorrect actions can cause serious operational harm

Question 9

Which statement best describes model drift?

A. A model’s source code is physically moved to another server

B. A model becomes less suitable as operating data or conditions change

C. A model is encrypted before deployment

D. A model is trained using only labelled data

Correct Answer: B. A model becomes less suitable as operating data or conditions change

Question 10

An organisation wants to prepare for future quantum threats. What should it do first?

A. Replace every system immediately without an inventory

B. Create a cryptographic inventory and identify long-lived sensitive data

C. Remove all encryption from archived data

D. Assume that symmetric encryption is already completely broken

Correct Answer: B. Create a cryptographic inventory and identify long-lived sensitive data