Artificial Intelligence and Information Security

Introduction

Artificial intelligence (AI) has become both a powerful instrument for defending information systems and a source of new security risk. Machine-learning systems can identify malicious activity at a scale and speed that exceed conventional rule-based controls. At the same time, attackers can manipulate training data, evade models, extract sensitive information, misuse generative AI, or compromise the infrastructure on which AI systems depend.

Information-security professionals therefore need more than a general understanding of machine learning. They must evaluate the security of AI models, data pipelines, deployment environments, human decisions, and organizational controls across the entire AI lifecycle. This chapter examines AI-enabled defense, adversarial machine learning, attacks against models and data, generative-AI risks, secure AI engineering, responsible use, governance, and human oversight.

AI security is a socio-technical discipline. Technical accuracy alone does not establish security: a highly accurate model may leak confidential data, produce unsafe decisions, or be vulnerable to carefully crafted inputs. Conversely, a less complex model may be preferable when it is explainable, auditable, robust, and easier to govern.

Learning Objectives

After studying this chapter, students should be able to:

1. Explain how AI supports information-security operations and identify its limitations.

2. Distinguish major adversarial machine-learning attacks against data, models, and systems.

3. Analyze the security and privacy risks of generative AI.

4. Apply secure-AI lifecycle principles to the design and deployment of an AI system.

5. Evaluate AI systems using security, reliability, privacy, fairness, and operational criteria.

6. Recommend governance arrangements that preserve human accountability and oversight.

1. AI in the Information-Security Landscape

AI refers broadly to computational systems that perform tasks associated with perception, reasoning, learning, prediction, generation, or decision support. Machine learning (ML) is a major AI approach in which a model learns relationships from data rather than relying exclusively on explicitly coded rules. Deep learning uses multilayer neural networks, while generative AI produces new text, code, images, audio, or other content from learned representations.

A security AI system usually includes five elements:

• Data sources: logs, network flows, endpoint events, identity records, threat intelligence, documents, or user reports.
• Data processing: collection, normalization, labeling, feature extraction, filtering, and storage.
• Model: a statistical or machine-learning artifact that classifies, predicts, ranks, detects, or generates.
• Decision and action layer: alerts, blocking, isolation, investigation, or automated response.
• Human and governance layer: analysts, policies, approvals, audit records, and accountability mechanisms.

This architecture creates an expanded attack surface. A security team must protect not only endpoints, networks, and applications but also training data, model parameters, prompts, application programming interfaces, model registries, feature stores, and inference infrastructure.

1.1 AI-enabled defensive applications

Common applications include:

• Anomaly detection: identifying behavior that deviates from a baseline, such as unusual login locations or abnormal data transfer.
• Malware and phishing analysis: classifying files, URLs, messages, and scripts using static, dynamic, and behavioral evidence.
• Security operations support: correlating alerts, summarizing incidents, recommending investigative queries, and prioritizing cases.
• Identity and access analytics: detecting impossible travel, credential misuse, privilege escalation, or account takeover.
• Vulnerability management: ranking weaknesses according to exploitability, asset criticality, exposure, and observed threat activity.
• Fraud detection: identifying suspicious transactions or account behavior.
• Incident response: assisting with containment recommendations, timeline construction, and evidence analysis.
• Security engineering: generating test cases, reviewing code, and identifying insecure configurations.

AI is particularly useful where the volume and velocity of events exceed human processing capacity. However, a model does not understand organizational context automatically. A rare but legitimate administrative action may resemble an attack, while a carefully designed attack may resemble normal activity.

1.2 Detection is not the same as security

A model can improve detection while increasing other risks. Important failure modes include:

• false positives that overwhelm analysts;
• false negatives that allow attacks to proceed;
• concept drift as attacker behavior and business processes change;
• biased or incomplete training data;
• automation bias, in which users accept model output without adequate verification;
• loss of evidence integrity if AI-generated summaries replace original records; and
• unsafe automated actions, such as disabling a critical account during an incident.

AI should therefore be evaluated as part of a control system rather than as an isolated prediction engine. The relevant question is not merely “How accurate is the model?” but “Does this system reduce risk under realistic operating conditions, including attack, uncertainty, failure, and human misuse?”

2. Security Properties and Evaluation Criteria

Traditional information security emphasizes confidentiality, integrity, and availability (CIA). AI systems require additional properties:

• Robustness: resistance to intentionally or unintentionally manipulated inputs.
• Reliability: consistent performance across relevant environments and time periods.
• Privacy: protection against disclosure or inference of sensitive information.
• Provenance: ability to establish where data, models, prompts, and outputs originated.
• Explainability: ability to provide information that helps an appropriate user understand a result.
• Fairness: avoidance of unjustified disparities among relevant groups.
• Accountability: assignment of responsibility for decisions and outcomes.
• Recoverability: ability to restore safe operation after model, data, or infrastructure compromise.

Security evaluation must be contextual. In a malware triage system, a false negative may be more costly than a false positive. In an identity-verification system, excessive false positives may deny legitimate access, while excessive false negatives may enable fraud. Thresholds should therefore be chosen using operational costs, not accuracy alone.

For binary classification, the following quantities are commonly used:

• True positives (TP): malicious cases correctly identified.
• True negatives (TN): benign cases correctly accepted.
• False positives (FP): benign cases incorrectly flagged.
• False negatives (FN): malicious cases missed by the system.

Precision is TP/(TP + FP), provided the denominator is nonzero. Recall, also called true-positive rate, is TP/(TP + FN), provided the denominator is nonzero. These measures describe different properties: precision concerns the credibility of alerts, while recall concerns the proportion of relevant threats detected. They must be interpreted with the base rate of attacks and the costs of errors. A model with high accuracy can still be operationally poor when attacks are rare and false positives dominate.

3. Adversarial Machine Learning

Adversarial machine learning studies how machine-learning systems behave when an adversary deliberately manipulates data, inputs, models, or operating conditions. Attacks can occur during training, inference, or deployment.

3.1 Evasion attacks

An evasion attack modifies an input at inference time so that the model produces an incorrect result. Examples include changing characteristics of a malicious file without destroying its harmful behavior, modifying network traffic patterns, or crafting a message that bypasses a phishing classifier.

A perturbation need not be visually obvious or large in a mathematical sense. What matters is whether it changes the model’s decision while preserving the attacker’s objective. Security systems should not assume that small changes are harmless: in high-dimensional feature spaces, a small input modification may cross a decision boundary.

Defenses include adversarial testing, robust feature engineering, input validation, model ensembles, behavior-based analysis, and independent controls. No single defense provides universal protection.

3.2 Poisoning attacks

A poisoning attack corrupts training or feedback data so that the resulting model behaves incorrectly. An attacker may insert mislabeled samples, manipulate telemetry, create deceptive reputation signals, or influence data used for continuous learning. Poisoning can reduce general performance or create a targeted weakness affecting selected inputs.

Controls include authenticated data collection, source validation, provenance records, access separation, approval for label changes, outlier analysis, robust training methods, and controlled retraining. Continuous-learning systems are especially sensitive because a compromised feedback loop can transform a short-lived intrusion into a persistent model defect.

3.3 Backdoors and trojaned models

A backdoored model behaves normally on ordinary inputs but produces attacker-chosen outputs when a secret trigger appears. The trigger may be a pattern in an image, a token sequence, a feature combination, or a particular operational condition. Backdoors may be introduced during training, model transfer, or supply-chain integration.

Model provenance, signed artifacts, reproducible builds, independent validation, trigger-oriented testing, and controlled model registries reduce this risk. A model should not be trusted merely because it was obtained from a popular repository or vendor.

3.4 Model extraction and inference attacks

In model extraction, an attacker queries a model and uses its outputs to approximate its decision function or recover intellectual property. Excessive query access, detailed confidence scores, and weak rate controls make extraction easier.

In membership inference, an attacker attempts to determine whether a particular record was included in training data. In model inversion, outputs are used to infer properties of training data or reconstruct representative inputs. These attacks demonstrate that a model can reveal information even when the original dataset is not directly exposed.

Mitigations include data minimization, privacy-preserving training, output restriction, access control, rate limiting, monitoring, and avoiding unnecessary disclosure of confidence information. Privacy risk depends on the data, model capacity, training procedure, and query environment; it cannot be eliminated simply by removing names from a dataset.

3.5 Availability and resource attacks

AI services may be attacked through query flooding, excessive prompt length, computationally expensive inputs, or repeated requests designed to increase inference cost. A denial-of-service attack against a model may affect a security operation just as seriously as an attack against a conventional application.

Capacity planning, quotas, authentication, workload isolation, queue management, timeouts, caching where safe, and graceful degradation are necessary operational controls.

4. Attacks on the AI Supply Chain and Lifecycle

An AI system inherits risks from its dependencies. The supply chain may include data brokers, open-source libraries, pretrained models, cloud services, annotation contractors, container images, hardware accelerators, and plugins.

Important risks include:

• malicious or manipulated training datasets;
• vulnerable preprocessing libraries;
• compromised model files;
• hidden dependencies and unpatched packages;
• unauthorized model replacement;
• insecure notebooks and development environments;
• leakage through experiment tracking systems;
• weak separation between development, testing, and production; and
• insufficient documentation of licensing and data rights.

A secure supply chain requires asset inventories, dependency scanning, artifact signing, software bills of materials, model cards, dataset documentation, secure build pipelines, vulnerability management, and independent acceptance testing. Access to training data and model repositories should follow least privilege and be logged.

5. Generative-AI Security Risks

Generative AI introduces risks beyond conventional classification and prediction. Large language models (LLMs), for example, generate outputs from statistical patterns and context. They do not guarantee truth, intent, confidentiality, or authorization.

5.1 Prompt injection

A prompt injection attempts to manipulate a model into disregarding intended instructions. Direct injection is supplied by the user; indirect injection is embedded in content the model retrieves, such as an email, web page, or document. This is especially dangerous when the model has tools or access to sensitive systems.

A secure design treats retrieved content as untrusted data, separates instructions from data, limits tool permissions, validates actions outside the model, and requires confirmation for high-impact operations. System prompts are not a substitute for access control.

5.2 Sensitive-information disclosure

Users may submit credentials, personal data, source code, confidential contracts, or incident details to an AI service. Information may be retained, logged, used for service improvement, exposed through misconfiguration, or reproduced in later outputs depending on the system architecture and provider terms.

Organizations need data-classification rules, approved tools, retention controls, encryption, tenant isolation, redaction, contractual review, and user training. Confidential data should not be sent to a service merely because it is convenient.

5.3 Hallucination and fabricated authority

A generative model can produce plausible but false statements, citations, commands, or code. In security operations, a fabricated indicator or incorrect remediation can waste resources or cause an outage. Outputs must therefore be treated as proposals requiring verification, particularly when they affect access, evidence, production systems, or legal decisions.

5.4 Insecure code and harmful content

AI-generated code may contain injection flaws, weak cryptography, insecure dependencies, or incorrect assumptions. Generated detection rules may be syntactically valid but operationally unsafe. Secure development practices—including code review, testing, static analysis, dependency scanning, and controlled deployment—remain necessary.

5.5 Excessive agency and tool abuse

An AI agent that can send messages, modify tickets, execute commands, or change access rights has a larger impact than a conversational assistant. Its permissions should be narrowly scoped, time-limited, observable, and revocable. High-impact actions should require human approval or independent policy enforcement.

6. A Secure AI Lifecycle

Security must be integrated from problem definition through retirement.

6.1 Govern and define the problem

The organization should identify the purpose, intended users, affected stakeholders, risk level, legal basis, success criteria, prohibited uses, and decision authority. A model should not be introduced simply because data and computing resources are available. The initial question is whether AI is appropriate at all.

6.2 Acquire and prepare data securely

Data governance includes lawful collection, purpose limitation, minimization, quality assessment, labeling controls, retention, access management, and provenance. Training and test data must be separated to reduce leakage. Sensitive attributes require special care because removing them may not remove their proxies.

6.3 Design and train

Threat modeling should consider poisoning, evasion, extraction, privacy leakage, supply-chain compromise, and misuse. Training environments should be isolated, secrets protected, dependencies controlled, and experiments recorded. Security requirements should be measurable, such as maximum acceptable false-negative rates for defined attack classes or maximum query rates for a public interface.

6.4 Validate and red-team

Testing should include ordinary performance, edge cases, distribution shifts, adversarial inputs, privacy probes, abuse scenarios, fairness analysis, and failure recovery. Red-teaming should examine both the model and the surrounding application. A model that resists adversarial examples in a laboratory may still fail because an exposed API, weak identity control, or unsafe tool integration bypasses the model.

6.5 Deploy with safeguards

Production deployment should use staged release, monitoring, rollback, model-version control, access restrictions, secure logging, and change approval. Automated actions should be proportional to confidence and impact. A low-confidence detection may create an investigation ticket, whereas account suspension may require corroborating evidence and human approval.

6.6 Monitor, update, and retire

Monitoring should cover model performance, data drift, concept drift, alert quality, resource consumption, access patterns, anomalous queries, and incidents. Retraining must be governed rather than automatic. When a model is retired, its data, credentials, endpoints, and downstream dependencies must be decommissioned securely.

7. Governance, Ethics, and Human Oversight

AI governance translates organizational values and legal duties into accountable processes. A useful governance structure assigns:

• a business owner responsible for purpose and risk acceptance;
• a technical owner responsible for implementation and performance;
• security personnel responsible for threat modeling and controls;
• privacy and legal specialists responsible for applicable obligations;
• independent reviewers responsible for assurance; and
• operational users responsible for appropriate interpretation and escalation.

Documentation should describe intended use, limitations, data sources, evaluation conditions, known failure modes, model version, dependencies, and approval status. Model cards and dataset documentation are useful communication mechanisms, although documentation does not replace testing.

Human oversight must be meaningful rather than ceremonial. A reviewer needs sufficient time, information, authority, and competence to reject or override a recommendation. If an operator is pressured to accept every model output, the nominal human-in-the-loop becomes a rubber stamp.

Ethical evaluation includes privacy, discrimination, autonomy, transparency, proportionality, and contestability. Security measures can themselves create social harm if surveillance is excessive or if automated suspicion is applied unequally. The appropriate objective is not maximum automation but trustworthy risk reduction.

Regulatory obligations differ across jurisdictions and sectors. Organizations should map AI practices to applicable privacy, cybersecurity, records, employment, financial, safety, and sector-specific rules. Widely used risk-management approaches emphasize governance, mapping, measurement, and management of AI risks. Such frameworks support disciplined practice but do not eliminate the need for legal interpretation or organizational judgment.

8. Illustrative Case: AI-Assisted Account-Takeover Detection

Background

A fictional online financial-service provider deploys an ML system to identify account takeover. The system analyzes login location, device characteristics, time patterns, authentication events, and transaction behavior. It assigns a risk score and recommends step-up authentication or temporary review.

Problem

After deployment, the system generates many alerts during a period of remote work and travel. At the same time, a criminal group begins using residential proxy services and previously unseen devices. The attackers also submit carefully shaped login patterns that resemble legitimate customers.

Analysis

The organization discovers three weaknesses. First, the training data overrepresented office-based users, creating a high false-positive rate for remote workers. Second, the model relied heavily on device and location features that the attackers could manipulate. Third, the response workflow automatically blocked accounts based on a single model score, creating customer harm and an attractive denial-of-service opportunity.

Approach and solution

The provider introduces a layered design. The model becomes one signal among several, including authentication strength, transaction context, known threat intelligence, and customer confirmation. High-impact actions require corroborating evidence. The data team expands evaluation sets to include travel, remote work, shared devices, accessibility scenarios, and proxy-based attacks. Security engineers conduct adversarial testing, while privacy staff review retention and use of location data.

The organization also implements model-version approval, drift monitoring, analyst feedback controls, rate limits, signed deployment artifacts, and a rollback procedure. Analysts receive explanations framed as contributing evidence rather than claims of certainty.

Outcome

False positives decline, and the system identifies more proxy-assisted attacks when combined with transaction and authentication signals. The organization accepts that some cases must be reviewed manually. It measures success using prevented losses, customer friction, analyst workload, false-negative investigations, and recovery time—not accuracy alone.

Learning points

1. A model’s training environment may not represent operational reality.

2. Features that are predictive are not necessarily difficult for attackers to manipulate.

3. High-impact automation requires proportionality and independent safeguards.

4. Human review is effective only when supported by evidence, authority, and usable workflows.

5. Security, privacy, fairness, and service quality must be evaluated together.

9. Practical Best Practices

1. Define the security purpose and unacceptable uses before selecting a model.

2. Classify data and prohibit unapproved disclosure to external AI services.

3. Maintain inventories of models, datasets, prompts, agents, dependencies, and interfaces.

4. Use least privilege for data, tools, model registries, and production actions.

5. Authenticate data sources and preserve provenance.

6. Test for poisoning, evasion, extraction, privacy leakage, prompt injection, and denial of service.

7. Separate model recommendations from authorization and policy enforcement.

8. Use independent controls for high-impact decisions.

9. Monitor drift, performance, access, cost, and unusual queries after deployment.

10. Document limitations and provide an operational rollback path.

11. Train users to recognize hallucinations, automation bias, and data-disclosure risks.

12. Review systems periodically and retire them when their risks exceed their benefits.

Challenges and Limitations

AI security remains difficult because attack techniques evolve, training data are imperfect, and security outcomes are often delayed or difficult to measure. Robustness methods can reduce performance or increase computational cost. Explainability techniques may describe correlations without establishing causal validity. Privacy-preserving methods can limit utility. Human oversight can fail through workload, inadequate expertise, or organizational pressure.

There is also no universal definition of a “secure model.” Security depends on the threat model, deployment architecture, data sensitivity, user population, and consequences of error. Benchmark performance cannot substitute for environment-specific testing. Finally, AI may amplify existing organizational weaknesses: unclear ownership, poor asset inventory, weak access controls, and inadequate incident response remain serious problems regardless of model sophistication.

Contemporary Developments

Current practice is moving toward multimodal models, retrieval-augmented generation, AI agents, confidential computing, privacy-enhancing machine learning, automated red-teaming, and continuous model monitoring. These developments may improve capability but also expand trust boundaries. Retrieval systems introduce document and embedding security concerns; agents introduce authorization and action-control concerns; multimodal systems introduce cross-modal prompt injection and content-verification challenges.

Research increasingly focuses on certified or empirically robust models, machine unlearning, provenance standards, secure model supply chains, privacy accounting, and evaluation of systemic risks. Professionals should follow standards, threat intelligence, and regulatory developments while recognizing that rapidly changing tools can outpace formal evidence.

Chapter Summary

AI can strengthen information security by detecting anomalies, prioritizing threats, supporting investigation, and automating selected defensive tasks. Its use also creates new attack surfaces involving data, models, interfaces, dependencies, and human decisions. Adversarial machine-learning threats include evasion, poisoning, backdoors, extraction, inference, and resource exhaustion. Generative AI adds prompt injection, sensitive-information disclosure, hallucination, insecure code, and excessive-agency risks.

Secure AI requires lifecycle governance: define an appropriate purpose, protect and document data, threat-model the system, validate it adversarially, deploy it with least privilege and monitoring, and maintain rollback and retirement procedures. Evaluation must consider robustness, privacy, reliability, fairness, explainability, accountability, and recovery—not accuracy alone. Human oversight must be competent and empowered, while organizational governance must assign responsibility and align technical practice with ethical and regulatory obligations.

Key Terms

Adversarial machine learning: Study of attacks and defenses involving deliberately manipulated machine-learning systems.

Anomaly detection: Identification of observations that differ materially from an expected pattern.

Backdoor: Hidden behavior causing a model to produce an attacker-controlled result when a trigger appears.

Concept drift: Change over time in the relationship between inputs and the target behavior.

Data poisoning: Manipulation of training or feedback data to degrade or control model behavior.

Evasion attack: Modification of an inference-time input to cause an incorrect model decision.

False negative: A malicious or relevant case that a system fails to identify.

False positive: A benign case incorrectly identified as malicious or relevant.

Generative AI: AI that produces new content such as text, code, images, or audio.

Hallucination: Plausible-looking but unsupported or incorrect generated content.

Human oversight: Supervision that enables a person to understand, challenge, approve, or override AI-supported decisions.

Model extraction: Approximation or theft of a model through queries or other observations.

Model inversion: Inferring sensitive characteristics or representative data from model behavior.

Model poisoning: See data poisoning; corruption intended to influence a trained model.

Prompt injection: Manipulation of instructions or retrieved content to alter a generative model’s behavior.

Provenance: Recorded history and origin of data, models, software, or outputs.

Retrieval-augmented generation: Generation assisted by retrieving external documents or knowledge sources.

Robustness: Ability to maintain acceptable behavior under perturbation, uncertainty, or attack.

Training-data leakage: Disclosure or inference of information used to train a model.

Trojaned model: A model containing concealed malicious behavior or an unauthorized trigger.

Review Questions

Short Answer Questions

1. Why is AI security broader than model accuracy?

2. Distinguish an evasion attack from a poisoning attack.

3. What is prompt injection, and why is it dangerous for AI agents?

4. Define model extraction and identify one mitigation.

5. Why is human oversight required for high-impact AI actions?

Long Answer Questions

1. Explain the principal ways AI can support security operations, and critically evaluate its limitations.

2. Describe a secure AI lifecycle from problem definition through retirement.

3. Compare evasion, poisoning, backdoor, membership-inference, and model-extraction attacks.

4. Analyze the security, privacy, and governance risks of using a generative-AI assistant in a security operations center.

5. Discuss why precision, recall, base rates, and operational costs must be considered together when evaluating a security model.

Analytical / Application Questions

1. An organization wants to automatically disable accounts whenever an AI model reports a risk score above a threshold. Design a safer decision architecture, including corroborating evidence, human review, monitoring, and rollback.

2. A language model can read internal documents and execute approved administrative commands. Identify the major attack paths and propose controls for identity, retrieval, prompts, tools, logging, and authorization.

3. A phishing classifier performs well on historical data but deteriorates after attackers change message style. Diagnose the likely causes and develop a plan for drift detection, adversarial testing, retraining, and governance.

Multiple Choice Questions with Answers

Question 1

Which attack modifies inference-time input so that a model produces an incorrect result?

A. Evasion attack

B. Poisoning attack

C. Model extraction

D. Backdoor insertion

Correct Answer: A. Evasion attack

Question 2

What is the principal security concern with a backdoored model?

A. It always produces random outputs

B. It behaves maliciously when a secret trigger appears

C. It cannot be updated after deployment

D. It requires no training data

Correct Answer: B. It behaves maliciously when a secret trigger appears

Question 3

Which measure is calculated as TP divided by TP plus FP?

A. Recall

B. Specificity

C. Precision

D. False-negative rate

Correct Answer: C. Precision

Question 4

Why can high accuracy be misleading for security classification?

A. Accuracy cannot be calculated for binary classifiers

B. Rare attacks may be obscured by many correctly classified benign cases

C. Accuracy always equals recall

D. Accuracy measures only computational cost

Correct Answer: B. Rare attacks may be obscured by many correctly classified benign cases

Question 5

Which control most directly reduces prompt-injection risk when an AI system uses retrieved documents?

A. Treat retrieved content as trusted system instructions

B. Give the model unrestricted administrative permissions

C. Separate untrusted content from instructions and validate tool actions externally

D. Disable all logging

Correct Answer: C. Separate untrusted content from instructions and validate tool actions externally

Question 6

What is the purpose of data provenance in an AI lifecycle?

A. To increase model complexity

B. To record the origin and history of data and artifacts

C. To eliminate the need for access control

D. To guarantee that predictions are correct

Correct Answer: B. To record the origin and history of data and artifacts

Question 7

Which situation best illustrates a poisoning attack?

A. An attacker floods a model API with requests

B. An attacker inserts manipulated examples into the training set

C. An attacker observes a model’s public output

D. A user forgets a password

Correct Answer: B. An attacker inserts manipulated examples into the training set

Question 8

Why should high-impact AI actions often require independent controls?

A. Models are incapable of producing any useful output

B. Independent controls reduce the consequences of model error or manipulation

C. Independent controls make monitoring unnecessary

D. They guarantee that false positives never occur

Correct Answer: B. Independent controls reduce the consequences of model error or manipulation

Question 9

Which property concerns protection against disclosure or inference of sensitive information?

A. Availability

B. Privacy

C. Latency

D. Throughput

Correct Answer: B. Privacy

Question 10

In the illustrative account-takeover case, why was a single model score insufficient for automatic blocking?

A. The model used no data at all

B. Attackers could manipulate some predictive features and legitimate users generated similar signals

C. The organization had no customers

D. The system was designed only for image generation

Correct Answer: B. Attackers could manipulate some predictive features and legitimate users generated similar signals