AI Security and Emerging Cloud Developments

Introduction

Artificial intelligence (AI) has become a cloud capability rather than a purely local software function. Cloud providers supply machine-learning platforms, managed model APIs, GPU infrastructure, data pipelines, vector databases, model registries, monitoring services, and generative-AI assistants. These services improve productivity, but they also introduce security risks that differ from conventional application security.

An AI system may expose confidential training data, generate inaccurate or harmful content, rely on vulnerable third-party models, or make decisions that are difficult to explain. Security therefore involves more than protecting servers and networks. It requires controls over data, models, prompts, identities, software dependencies, human use, automated actions, and the legal environment in which the system operates.

This chapter examines cloud-hosted AI and related developments: generative-AI threats, model and data attacks, shadow AI, automation, confidential computing, post-quantum cryptography, edge cloud, sovereign cloud, and evolving regulatory and industry requirements.

Learning Objectives

By the end of this chapter, learners should be able to:

1. Explain the security properties and attack surfaces of cloud-hosted and generative-AI systems.
2. Distinguish attacks against data, models, prompts, tools, identities, and infrastructure.
3. Assess risks associated with shadow AI and excessive automation.
4. Describe the security value and limitations of confidential computing.
5. Explain why post-quantum cryptography matters to cloud security.
6. Compare edge cloud and sovereign cloud from a risk-management perspective.
7. Apply a practical governance framework to an AI-enabled cloud service.
8. Discuss ethical, regulatory, and professional responsibilities in deploying AI.

1. The Cloud AI Security Landscape

A conventional cloud application usually consists of software, infrastructure, data, identities, and interfaces. A cloud AI system adds several specialized assets:

• Training data, which may contain personal, confidential, copyrighted, or regulated information.
• Foundation models, supplied by a provider and often shared across many customers.
• Fine-tuned models, adapted to an organization’s data or tasks.
• Prompts and context, which can reveal business information or control model behaviour.
• Embeddings and vector indexes, which support semantic search and retrieval-augmented generation.
• Tools and agents, which allow a model to call APIs, execute workflows, or change records.
• Evaluation and monitoring data, which may itself contain sensitive user interactions.

The security objectives remain confidentiality, integrity, and availability, but they require interpretation in an AI context. Confidentiality includes preventing unauthorized disclosure through prompts, outputs, logs, embeddings, or model inversion. Integrity includes ensuring that training data, model parameters, retrieved documents, and automated decisions have not been manipulated. Availability includes resilience of model APIs, GPU capacity, retrieval systems, and dependent services.

AI also introduces reliability and safety concerns. A model can be secure against unauthorized access yet still produce unsafe or misleading results. Security engineering must therefore be complemented by validity testing, human oversight, and operational controls.

1.1 Shared responsibility for cloud AI

The cloud provider commonly secures physical facilities, core hardware, and managed service infrastructure. The customer remains responsible for configuration, identities, data classification, prompts, access permissions, application logic, and appropriate use. Responsibility varies by service model.

Contracts and technical documentation must be examined because data retention, provider training use, geographic processing, logging, and support access differ between services.

2. Generative AI Threats

Generative AI produces text, code, images, audio, or other content from an input prompt. Its security risks arise from both the model and the surrounding application.

Prompt injection

A prompt injection attempts to influence a model to ignore intended instructions, disclose protected context, or perform an unauthorized action. A direct injection is placed in the user’s prompt. An indirect injection is hidden in content retrieved by the system, such as a web page, email, document, or issue ticket. The latter is especially important in retrieval-augmented generation (RAG), because the model may treat retrieved text as instructions rather than untrusted data.

Prompt filtering alone is insufficient. Stronger controls include separating system instructions from retrieved content, marking content as untrusted, limiting the tools available to the model, validating tool parameters, requiring approval for high-impact actions, and applying ordinary authorization checks outside the model.

Data leakage and insecure output handling

A model may reproduce sensitive information from its context, conversation history, logs, or training material. Output can also become a security input. For example, generated code may be copied into production, or generated text may be passed into an SQL query, shell command, HTML page, or email system without validation. The application must treat model output as untrusted input and apply encoding, schema validation, sandboxing, and least privilege.

Hallucination and over-reliance

A hallucination is a plausible but unsupported output. It is not necessarily a malicious attack, but it becomes a security risk when an organization relies on it for legal advice, medical decisions, financial approval, incident response, or code deployment. Mitigations include retrieval from authoritative sources, citations, confidence-aware workflows, testing against representative cases, and human review for consequential decisions.

Model denial of service and cost abuse

Attackers may submit unusually long prompts, repeated requests, or computationally expensive inputs. This can exhaust quotas, increase cloud costs, or reduce availability. Rate limiting, quotas, request-size limits, workload isolation, anomaly detection, and budget alerts are appropriate controls.

3. Attacks Against Data and Models

Data poisoning

Data poisoning occurs when an attacker inserts misleading, malicious, or biased examples into training, fine-tuning, evaluation, or retrieval data. A poisoned dataset can cause systematic incorrect outputs or create a trigger that activates specific behaviour. Controls include provenance tracking, trusted ingestion pipelines, duplicate and anomaly detection, review of high-impact data, versioned datasets, and the ability to reproduce and roll back training.

Training-data extraction and membership inference

A model may unintentionally reveal information about examples used during training. Training-data extraction attempts to recover memorized content. Membership inference attempts to determine whether a particular record was included in training. Risk depends on data sensitivity, model design, training procedure, access, and memorization. Data minimization, de-identification where appropriate, contractual restrictions, privacy-preserving training, and output monitoring reduce exposure.

Model theft and model extraction

An attacker can query a model repeatedly to approximate its behaviour or recover intellectual property. Excessive access, unrestricted APIs, and weak tenant isolation increase the risk. Defences include authentication, quotas, abuse detection, output controls, watermarking or provenance mechanisms where suitable, and separation of sensitive model endpoints from public interfaces.

Adversarial examples and evasion

Small, deliberately chosen changes to an input may cause an AI classifier to misclassify it. In cloud environments this may affect fraud detection, malware analysis, biometric systems, or image inspection. Robustness testing should include realistic attacks, distribution shifts, and failure-safe responses. No model should be treated as the sole security boundary for a high-impact decision.

Supply-chain compromise

AI systems depend on datasets, open-source libraries, pretrained models, container images, plugins, and hosted APIs. A compromised model file or package may contain malicious code, a backdoor, or an unsafe dependency. Organisations should maintain a software and model bill of materials, verify signatures and hashes, scan dependencies, pin versions, restrict model loading, and assess suppliers.

4. Shadow AI and Automation Risk

Shadow AI is the use of AI tools without formal approval, inventory, or security assessment. Employees may paste source code, customer records, contracts, or incident details into public services. They may also install browser extensions, connect personal accounts, or use unapproved plugins.

A prohibition without usable alternatives is unlikely to work. A responsible programme should provide approved tools, clear data-handling rules, training, procurement review, monitoring consistent with privacy law, and a reporting route for accidental disclosure. Data-loss-prevention controls can identify sensitive patterns, but they should be tested to avoid excessive blocking and false alarms.

Automation magnifies both benefits and errors. An AI agent with read-only access has a different risk profile from one that can delete records, issue refunds, deploy code, or change identity permissions. A useful design principle is graduated autonomy:

1. Observe: the system collects information but takes no action.
2. Recommend: it proposes an action for a person.
3. Execute with approval: it performs an approved action.
4. Execute within limits: it acts automatically under explicit policy constraints.
5. Fully autonomous operation: reserved for low-impact, reversible, well-tested tasks.

Every action should be authenticated, authorized, logged, attributable to a human or service identity, and reversible where practical. Tool access should be narrowly scoped, with transaction limits and separation of duties.

5. A Practical AI Risk Model

A simple qualitative model can help prioritize controls. For a threat scenario, risk is assessed from likelihood and impact. A numerical score can be used only as a decision aid, not as an objective measurement.

If both dimensions are rated from 1 to 5, the score ranges from 1 to 25. Likelihood may consider exposure, attacker capability, control strength, and event frequency. Impact may consider confidentiality, integrity, availability, safety, legal consequences, financial loss, and reputation. The ratings are ordinal; multiplying them does not prove that risk is measured on a true ratio scale. Therefore, the organization should document rating definitions and use professional judgement.

A more useful AI risk assessment records the following for each use case:

The NIST AI Risk Management Framework, the NIST Cybersecurity Framework, ISO/IEC 27001, ISO/IEC 42001, and sector-specific rules can provide complementary governance structures. They should be adapted to the organization rather than treated as substitutes for technical analysis.

6. Confidential Computing

Confidential computing protects data in use by executing workloads inside a hardware-supported trusted execution environment (TEE). Traditional encryption commonly protects data at rest and in transit; confidential computing aims to reduce exposure while data is being processed in memory.

A TEE typically provides isolation, measured boot, and attestation. Attestation allows a relying party to verify that an approved software measurement is running in an expected environment before releasing a key or data. This can be valuable when processing sensitive AI prompts, proprietary models, or regulated datasets on shared cloud infrastructure.

However, confidential computing is not a complete solution. It does not automatically protect data before entry or after release, prevent an authorized application from misusing data, eliminate side-channel risks, or guarantee that model outputs are correct. Key management, attestation policy, patching, workload design, and monitoring remain essential. Performance and hardware-support limitations may also affect GPU-intensive AI workloads.

7. Post-Quantum Cryptography

A sufficiently capable quantum computer could threaten widely used public-key systems based on integer factorization or discrete logarithms, such as RSA and elliptic-curve cryptography. Symmetric cryptography is generally less severely affected, although security margins may need review.

The main practical concern is harvest now, decrypt later: an adversary may collect encrypted traffic today and attempt decryption when cryptanalysis capabilities improve. Long-lived cloud data, certificates, software-signing systems, and confidential research are especially relevant.

Post-quantum cryptography (PQC) uses algorithms designed to resist known quantum attacks. Standards work by NIST has selected algorithms including ML-KEM for key establishment and ML-DSA for digital signatures; organizations should follow current standards and provider guidance because implementation profiles and approved algorithms may evolve.

Migration requires a cryptographic inventory, identification of data with long confidentiality lifetimes, supplier engagement, protocol testing, certificate planning, and crypto-agility. Hybrid approaches may combine classical and post-quantum mechanisms during transition. PQC can increase key and signature sizes, creating bandwidth, storage, latency, and compatibility impacts. It should therefore be planned as an engineering programme rather than installed as a single product.

8. Edge Cloud and Distributed AI

Edge cloud places computing, storage, or AI inference closer to users, devices, or operational environments. It can reduce latency, limit the transfer of raw data, and support operation during intermittent connectivity. Examples include retail analytics, industrial monitoring, connected vehicles, and remote healthcare devices.

The distributed architecture expands the attack surface. Edge nodes may be physically accessible, intermittently connected, less frequently patched, and heterogeneous. Controls include secure boot, device identity, signed updates, encrypted communications, local access restrictions, tamper detection, segmentation, remote attestation where appropriate, and centralized inventory. Data synchronization must address replay, duplication, conflict, and loss.

Processing data locally may improve privacy but does not automatically make processing lawful or secure. Edge devices can still leak data, generate unsafe inferences, or be compromised and used as a route into cloud services.

9. Sovereign Cloud and Data Residency

A sovereign cloud seeks to provide stronger national or jurisdictional control over data, operations, personnel, infrastructure, and legal access. It may be relevant to public administration, defence, critical infrastructure, and regulated industries.

Data residency means data is stored or processed in a specified location. Data sovereignty concerns the laws and authority that may apply to data. These are related but not identical. Data stored in a country may still be accessible to a foreign-controlled provider under applicable legal arrangements. A sovereignty assessment should examine ownership, provider control, administrator location, support access, subcontractors, encryption-key control, service continuity, and government-access obligations.

Sovereign solutions can reduce some jurisdictional risks but may reduce provider choice, geographic redundancy, or access to the newest AI models. They should be evaluated against explicit business and legal requirements rather than treated as automatically safer.

10. Regulation, Ethics, and Industry Expectations

AI governance is developing through legislation, regulatory guidance, standards, procurement rules, and professional practice. Requirements vary by jurisdiction and sector. Common expectations include lawful processing, transparency, human oversight, risk management, security, record keeping, impact assessment, incident reporting, and rights for affected individuals.

The European Union AI Act, for example, uses a risk-based approach with obligations that differ according to the AI system and its use. It should not be assumed that every generative-AI use has identical legal duties. The General Data Protection Regulation may apply when personal data is processed, including requirements concerning lawful basis, security, data-subject rights, and high-risk processing assessments. Other jurisdictions have their own laws and guidance.

Ethical security practice includes:

• Fairness: testing for unjustified differences in error rates or outcomes.
• Accountability: assigning an identifiable owner for system behaviour.
• Transparency: explaining purpose, limitations, and significant decisions in understandable language.
• Privacy: collecting and retaining only what is necessary.
• Human agency: allowing meaningful review, challenge, and appeal where decisions affect people.
• Accessibility: considering users with different abilities, languages, and levels of digital access.
• Environmental responsibility: considering energy and hardware demands of large-scale AI.

Compliance is a baseline, not proof of safety. Evidence should include threat models, data-flow diagrams, model cards or system documentation, evaluation results, access records, supplier assessments, incident exercises, and documented decisions.

11. Illustrative Case: AI-Assisted Insurance Claims

Background

An insurer deploys a cloud AI service to summarize claims, retrieve policy clauses, identify missing evidence, and recommend a priority level. The system connects to a document store and a claims-management API.

Problem

An employee uploads a claim containing medical information to an unapproved public chatbot to obtain a quick summary. Separately, a malicious sentence hidden in an uploaded document instructs the RAG system to reveal internal policy text. The model then recommends a high priority for claims containing certain demographic terms, although those terms are not relevant to urgency.

Analysis

The scenario contains shadow AI, sensitive personal data, indirect prompt injection, excessive retrieval permissions, and potential discriminatory impact. The model recommendation is not itself an authorized insurance decision, but staff may over-rely on it. The API connection also creates a risk that a model-generated instruction could alter a claim.

Approach and solution

The insurer establishes an approved AI gateway with tenant isolation, encryption, retention limits, and data-loss-prevention checks. Staff receive an enterprise assistant that prevents submission of restricted data to unapproved services. Retrieved documents are labelled as untrusted content; the model cannot treat them as system instructions. The claims API is read-only for the initial release, and any change requires a human approval step.

The organization creates a representative evaluation set, tests error rates across relevant groups, records citations for retrieved policy clauses, and requires staff to verify recommendations. Access is logged using individual identities. Supplier contracts address processing locations, breach notification, deletion, subprocessors, and model-training use. A rollback plan disables the AI endpoint without stopping ordinary claims processing.

Outcome

The system remains useful for summarization and evidence checking while high-impact decisions remain reviewable. The organization also gains an inventory of AI use, an incident process, and measurable evidence for internal audit and regulatory enquiries.

Learning points

Security depends on the complete workflow, not merely the model’s accuracy. Least privilege, approved alternatives, input and output controls, evaluation, human oversight, and recoverability must operate together.

12. Implementation Guidance for Cloud Practitioners

A diploma-level practitioner should be able to contribute to the following control cycle:

1. Inventory: record models, datasets, endpoints, agents, providers, regions, and owners.
2. Classify: label data and identify regulated, confidential, safety-critical, or public content.
3. Threat-model: consider prompt injection, poisoning, leakage, extraction, denial of service, supply chain, insider misuse, and unsafe automation.
4. Design access controls: use least privilege, separate tenants, short-lived credentials, network segmentation, and explicit tool permissions.
5. Protect the pipeline: verify data provenance, scan dependencies, sign artifacts, version models, and maintain rollback capability.
6. Evaluate: test accuracy, robustness, fairness, privacy leakage, abuse resistance, and failure handling before production.
7. Monitor: collect privacy-conscious logs for prompts, outputs, tool calls, model versions, identities, latency, cost, and policy violations.
8. Respond: define playbooks for leakage, poisoning, compromised suppliers, harmful output, unexpected cost, and unauthorized action.
9. Review: reassess after model changes, data changes, new integrations, legal changes, incidents, and material shifts in use.

Controls should be proportionate. A low-risk internal drafting assistant does not require the same approval process as an autonomous system affecting employment, credit, healthcare, safety, or public services. Nevertheless, every system needs an owner, an inventory entry, a defined purpose, and a route for reporting problems.

Challenges and Limitations

AI security is difficult because model behaviour is probabilistic, datasets change, providers may not reveal internal training details, and attacks evolve rapidly. Testing a finite set of prompts cannot prove that a model is safe. Privacy techniques may reduce utility; strict isolation may increase cost; sovereignty controls may reduce resilience; and post-quantum migration may create compatibility problems. Human review can itself be inconsistent or biased. Finally, standards and regulations continue to develop, so governance must support revision rather than rely on a one-time certification.

Chapter Summary

Cloud-hosted AI expands the security boundary to include data, models, prompts, embeddings, tools, suppliers, and human workflows. Generative-AI threats include prompt injection, data leakage, insecure output handling, hallucination, denial of service, poisoning, model extraction, adversarial inputs, and supply-chain compromise. Shadow AI creates unmanaged disclosure and compliance risks, while automation requires graduated autonomy, least privilege, approval gates, logging, and reversibility.

Confidential computing can protect data in use through trusted execution environments and attestation, but it does not replace application security or governance. Post-quantum cryptography requires inventory, prioritization, crypto-agility, and planned migration. Edge cloud improves latency and may reduce data transfer but creates distributed-device risks. Sovereign cloud addresses jurisdictional and operational control concerns, but residency and sovereignty are not identical.

Effective AI security combines technical safeguards, risk assessment, supplier governance, ethical review, regulatory awareness, continuous evaluation, and incident readiness. Compliance is evidence-based and contextual; it is not a guarantee that an AI system is safe or fair.

Key Terms

AI security: Protection and responsible governance of AI systems, their data, models, interfaces, and operations.

Generative AI: AI that creates content such as text, code, images, audio, or video.

Prompt injection: An attempt to manipulate model instructions through direct or retrieved input.

Indirect prompt injection: Prompt manipulation embedded in external content consumed by a model.

Retrieval-augmented generation (RAG): A design that retrieves external information and supplies it to a generative model.

Data poisoning: Deliberate corruption of training, fine-tuning, evaluation, or retrieval data.

Model extraction: Reconstructing or approximating a model through repeated queries.

Membership inference: Attempting to determine whether a record was used in model training.

Shadow AI: Unapproved or unrecorded use of AI tools within an organization.

Graduated autonomy: Increasing automation according to task impact, reversibility, and confidence.

Confidential computing: Protection of data during processing using hardware-supported trusted environments.

Trusted execution environment (TEE): An isolated environment designed to protect code and data during execution.

Attestation: Evidence that approved software is running in an expected protected environment.

Post-quantum cryptography (PQC): Cryptography designed to resist attacks from capable quantum computers.

Crypto-agility: The ability to replace cryptographic algorithms without major system redesign.

Edge cloud: Cloud-like processing and storage placed near users, devices, or data sources.

Data residency: The location where data is stored or processed.

Data sovereignty: The legal and governmental authority applicable to data and its processing.

Model supply chain: The datasets, models, libraries, tools, containers, and providers on which an AI system depends.

AI governance: Policies, responsibilities, controls, and evidence used to manage AI risks and impacts.

Review Questions

5 Short Answer Questions

1. What is the difference between direct and indirect prompt injection?
2. Why is model output treated as untrusted input?
3. State two limitations of confidential computing.
4. What is meant by shadow AI?
5. Distinguish data residency from data sovereignty.

5 Long Answer Questions

1. Explain the shared-responsibility model for a managed cloud AI service.
2. Discuss the main attacks against AI data and models and corresponding controls.
3. Explain how graduated autonomy reduces the risks of AI agents.
4. Describe the stages of a practical AI risk-management cycle.
5. Compare the security benefits and limitations of edge cloud and sovereign cloud.

3 Analytical / Application Questions

1. An AI agent can read customer records and issue refunds up to $500. Design an authorization, approval, logging, and rollback model for this agent.
2. A company plans to send ten-year confidential research archives to a cloud provider. Analyse the relevance of harvest-now-decrypt-later risk and propose a PQC migration plan.
3. A public agency wants to use a generative model to prioritize service applications. Identify privacy, fairness, security, transparency, and human-oversight risks, and recommend evidence that should be collected before deployment.

Multiple Choice Questions with Answers

Question 1

Which control most directly reduces the risk of an AI agent making an unauthorized high-impact change?

A. Increasing the model temperature

B. Requiring human approval and narrowly scoped tool permissions

C. Removing all audit logs

D. Allowing unrestricted API access

Correct Answer: B. Requiring human approval and narrowly scoped tool permissions

Question 2

What is an indirect prompt injection?

A. A hardware failure in a GPU

B. A prompt embedded in retrieved or external content

C. A cryptographic key rotation

D. A model version upgrade

Correct Answer: B. A prompt embedded in retrieved or external content

Question 3

What is the primary purpose of confidential computing?

A. To guarantee that model outputs are correct

B. To protect data while it is being processed

C. To eliminate the need for access control

D. To prevent all denial-of-service attacks

Correct Answer: B. To protect data while it is being processed

Question 4

Which activity is an example of data poisoning?

A. Adding malicious examples to a fine-tuning dataset

B. Encrypting a dataset at rest

C. Rotating an API credential

D. Compressing model weights

Correct Answer: A. Adding malicious examples to a fine-tuning dataset

Question 5

Why is post-quantum cryptography important to long-lived cloud data?

A. Quantum computers increase storage capacity

B. Attackers may collect encrypted data now and decrypt it later

C. It eliminates the need for symmetric encryption

D. It guarantees data residency

Correct Answer: B. Attackers may collect encrypted data now and decrypt it later

Question 6

Which statement correctly distinguishes data residency from data sovereignty?

A. They are always identical concepts

B. Residency concerns location, while sovereignty concerns applicable legal authority

C. Sovereignty concerns only network latency

D. Residency applies only to public data

Correct Answer: B. Residency concerns location, while sovereignty concerns applicable legal authority

Question 7

Which practice best addresses shadow AI?

A. Providing approved tools together with policy, training, and monitoring

B. Ignoring employee use of public tools

C. Allowing staff to submit any data to any service

D. Removing all AI services from the organization

Correct Answer: A. Providing approved tools together with policy, training, and monitoring

Question 8

Why should model-generated code be treated as untrusted?

A. It is always encrypted

B. It may contain errors or unsafe instructions and can affect downstream systems

C. It cannot be copied into applications

D. It is guaranteed to be unbiased

Correct Answer: B. It may contain errors or unsafe instructions and can affect downstream systems

Question 9

Which characteristic is a common security challenge of edge cloud?

A. All processing occurs in one centrally controlled facility

B. Edge devices may be physically accessible and inconsistently patched

C. Edge systems cannot process data locally

D. Edge systems remove the need for device identity

Correct Answer: B. Edge devices may be physically accessible and inconsistently patched

Question 10

In the qualitative model Risk score = Likelihood × Impact, what is the score when likelihood is 4 and impact is 5?

A. 9

B. 16

C. 20

D. 25

Correct Answer: C. 20