Human Factors, AI Security and Emerging Trends

Introduction

Cyber security is often described as a technical discipline involving networks, software, cryptography and hardware. In practice, however, security outcomes depend heavily on people, organisational processes and the rapidly changing technologies used by both defenders and attackers. A correctly configured firewall cannot prevent an employee from disclosing a password to a convincing impersonator. Similarly, an artificial-intelligence system may improve detection while also generating persuasive phishing messages, discovering vulnerabilities or enabling large-scale fraud.

This chapter examines three connected areas. First, it considers human factors and the design of security awareness and phishing defences. Second, it explains defensive and malicious applications of artificial intelligence (AI), including adversarial machine learning and generative-AI risks. Third, it introduces important emerging trends: post-quantum cryptography, software supply-chain threats and changing workforce requirements. The central argument is that effective cyber security is socio-technical: technology, people, governance and operational practice must be designed together.

Learning Objectives

By the end of this chapter, a student should be able to:

• explain why human behaviour is a security control and a security risk;
• design a practical security-awareness and phishing-resistance programme;
• distinguish automation, machine learning, generative AI and adversarial machine learning;
• describe defensive and malicious uses of AI;
• identify prompt-injection, data-poisoning, model-extraction and deepfake risks;
• explain the purpose and transition challenges of post-quantum cryptography;
• analyse software and hardware supply-chain threats;
• recommend governance, technical and workforce controls for emerging risks.

1. Human Factors in Cyber Security

1.1 The socio-technical nature of security

A security control operates within a human environment. Users interpret warnings, choose passwords, approve access requests, report incidents and decide whether a process is trustworthy. Attackers exploit these decisions through social engineering, which is the manipulation of people to obtain information, access or action.

Human error is not a single behaviour. It can include:

• slips, such as sending a file to the wrong recipient;
• mistakes, such as incorrectly believing that a message is genuine;
• violations, such as deliberately bypassing a security procedure;
• inattention, caused by workload, interruptions or alert fatigue;
• unsafe adaptation, where staff create informal workarounds because a control is impractical.

A blame-oriented response often reduces reporting. If employees expect punishment for admitting an error, they may conceal incidents, increasing the time available to an attacker. A mature organisation uses accountability for reckless or malicious behaviour but treats ordinary mistakes as opportunities for process improvement.

1.2 Why people respond to phishing

Phishing messages commonly exploit authority, urgency, fear, curiosity, financial incentive and familiarity. A message that appears to come from a manager may request an urgent payment. A fake cloud-storage notification may ask the user to re-authenticate. A targeted attack, known as spear phishing, uses information about a particular person or organisation. Business email compromise frequently combines impersonation with payment or invoice fraud.

Users do not make decisions in a vacuum. Relevant influences include:

• time pressure and competing tasks;
• trust in organisational authority;
• interface design and visual similarity to genuine services;
• previous experience with legitimate urgent requests;
• fatigue and repeated security warnings;
• the perceived cost of reporting a suspicious message.

Security awareness should therefore teach decision-making and provide usable reporting mechanisms, rather than merely instructing users to “be careful.”

2. Designing Security Awareness and Phishing Defences

2.1 A layered awareness programme

A professional awareness programme combines education, technical controls, simulation, measurement and feedback. A suitable cycle is:

1. Assess the organisation’s risks, roles, incidents and regulatory obligations.
2. Design role-specific learning for general users, privileged administrators, finance staff, developers and executives.
3. Deliver short, accessible learning using examples relevant to the organisation.
4. Reinforce learning through reminders, exercises and secure defaults.
5. Measure behaviour and outcomes, not merely course completion.
6. Improve content and controls using incident and exercise evidence.

Training should explain how to inspect the sender domain, verify unusual requests through an independent channel, avoid entering credentials through unexpected links, handle attachments safely and report suspected compromise quickly. It must also teach that a familiar logo, grammatical quality or presence of a name does not prove authenticity.

2.2 Technical and procedural controls

Awareness is strongest when supported by technology. Important controls include:

• phishing-resistant multifactor authentication, especially hardware-backed passkeys or security keys;
• secure email gateways, attachment sandboxing and malicious-link analysis;
• sender authentication using SPF, DKIM and DMARC, correctly configured for the organisation’s domains;
• browser and endpoint protections;
• least privilege and conditional access;
• payment-change verification using an approved second channel;
• simple reporting buttons and rapid incident-response procedures;
• password managers and prohibition of password reuse.

No single control is sufficient. Multifactor authentication reduces the value of stolen passwords, but attackers may still use session theft, social engineering or fraudulent approval requests. Email filtering reduces exposure, but carefully crafted messages can evade automated detection. Layering controls reduces dependence on any one assumption.

2.3 Phishing simulations and ethical limits

Simulated phishing can identify weaknesses, but poorly designed exercises damage trust. Simulations should have a documented purpose, proportionate difficulty and clear safeguards. They should not humiliate individuals, collect real passwords or create unnecessary distress. Results should primarily guide training and process improvement, with access to personally identifiable results restricted to authorised personnel.

Useful measures include:

A useful principle is to reward reporting, including reports of messages later found to be legitimate. False positives generally cost less than undetected compromise, provided the reporting process is efficient.

3. Artificial Intelligence in Cyber Security

3.1 Terminology

Artificial intelligence refers broadly to computational systems performing tasks associated with human intelligence, such as classification, prediction, language processing or decision support. Machine learning is a set of methods in which a system learns patterns from data rather than relying only on explicitly programmed rules. Deep learning uses multilayer neural networks. Generative AI creates new text, images, audio, video, software or other content from learned patterns.

AI does not possess guaranteed understanding or truthfulness. A model may produce a fluent but incorrect answer, a phenomenon commonly called hallucination. Security decisions must therefore include validation, accountability and appropriate human oversight.

3.2 Defensive uses

Defensive applications include:

• detecting anomalous logins, unusual data transfers and malware behaviour;
• classifying spam, phishing and malicious attachments;
• prioritising vulnerabilities according to exposure and likely impact;
• assisting security-operations-centre analysts with alert summarisation;
• generating detection rules, queries and incident-response drafts;
• analysing malware behaviour in controlled environments;
• identifying exposed secrets or insecure code during development;
• supporting threat-intelligence correlation and fraud detection.

AI is especially useful where the volume of events exceeds human capacity. Nevertheless, it should normally recommend, prioritise or assist rather than independently perform high-impact actions without safeguards. A false positive can interrupt essential services; a false negative can allow compromise.

A basic classification vocabulary is important. A true positive is a correctly identified malicious event, while a false positive is a benign event incorrectly labelled malicious. A true negative is a correctly identified benign event, and a false negative is a missed malicious event. In security operations, the cost of false negatives and false positives differs by context. Blocking a hospital’s legitimate authentication may be more harmful than delaying a low-risk transaction, whereas missing ransomware activity may be catastrophic.

3.3 Malicious uses of AI

Attackers can use AI to increase scale, speed and personalisation. Examples include:

• generating convincing multilingual phishing and business-email-compromise messages;
• automating reconnaissance and adaptation of social-engineering campaigns;
• producing polymorphic or rapidly changing malicious code;
• creating voice cloning and deepfake video for impersonation;
• searching public data for targets and persuasive contextual details;
• accelerating vulnerability research and password-guessing workflows.

AI does not eliminate the need for attacker knowledge. Generated output may be inaccurate, detectable or operationally unsafe. However, lower barriers to producing convincing content can increase the number and quality of attacks faced by ordinary users.

4. Adversarial and Generative-AI Risks

4.1 Adversarial machine learning

Adversarial machine learning studies attacks against systems that learn from data. An evasion attack modifies an input at inference time so that a model makes an incorrect decision; for example, a malicious file may be altered to avoid classification. A poisoning attack corrupts training data so that the trained model learns harmful patterns. A backdoor attack inserts a trigger that causes a particular malicious output when present. Model extraction attempts to reproduce a model or infer its decision behaviour through queries. Membership inference attempts to determine whether a particular record was included in training data.

Defences include trusted data pipelines, provenance records, dataset validation, outlier analysis, access controls, rate limiting, robust evaluation and monitoring for distribution shift. These controls reduce risk but cannot guarantee perfect robustness. A model trained and tested only on historical data may fail when attackers change tactics or when the operational environment changes.

4.2 Generative-AI security risks

Generative-AI systems introduce both conventional and distinctive risks:

• prompt injection: untrusted text attempts to override instructions or manipulate a model;
• indirect prompt injection: malicious instructions are embedded in retrieved documents, websites or emails;
• data leakage: confidential prompts, retrieved records or model outputs expose sensitive information;
• insecure tool use: an AI agent is induced to send email, execute code or modify records improperly;
• hallucination: fabricated facts or citations lead to unsafe decisions;
• copyright, privacy and provenance concerns: training or generated content may be legally or ethically problematic;
• overreliance and automation bias: users accept plausible output without verification;
• model supply-chain risk: a downloaded model, plug-in or dataset contains malicious code or a backdoor.

A secure deployment should classify data before it is sent to a model, minimise retained prompts, separate trusted instructions from untrusted content, restrict tools using least privilege, require confirmation for consequential actions, log interactions and test the system against realistic abuse cases.

4.3 Governance of AI systems

AI governance should identify the system owner, purpose, data sources, affected stakeholders, acceptable uses, prohibited uses, monitoring requirements and escalation process. A risk assessment should consider confidentiality, integrity, availability, privacy, fairness, safety and accountability. Human review is particularly important for employment, education, financial, healthcare, law-enforcement and access-control decisions.

Organisations should maintain an inventory of AI systems, including informal employee use of public tools. Policies should distinguish approved enterprise services from unapproved services and should specify what data may be entered. Technical controls such as data-loss prevention, identity management and audit logging should support—not replace—clear policy.

5. Post-Quantum Cryptography

5.1 The emerging threat

Current public-key systems such as RSA and elliptic-curve cryptography rely on mathematical problems that are difficult for conventional computers. A sufficiently capable quantum computer running Shor’s algorithm could undermine the security assumptions of widely used public-key encryption and digital signatures. Symmetric cryptography is affected differently; quantum search techniques provide a speedup against exhaustive search, so appropriately selected longer keys can provide substantial protection.

The threat is relevant before a cryptographically capable quantum computer exists because of “harvest now, decrypt later.” An adversary may capture encrypted traffic today and attempt decryption in the future. Long-lived secrets, government records, intellectual property and health data are particularly important.

5.2 Post-quantum approach and migration

Post-quantum cryptography (PQC) uses algorithms designed to resist known quantum and classical attacks while running on conventional computers. Important families include lattice-based, hash-based, code-based and other structured mathematical approaches. Standardisation and implementation profiles continue to evolve; organisations should follow authoritative standards and validated libraries rather than inventing algorithms.

Migration is an engineering and governance project:

1. inventory cryptographic algorithms, certificates, protocols and key lifetimes;
2. identify systems whose data must remain confidential for many years;
3. assess vendor roadmaps and hardware or protocol constraints;
4. test interoperable PQC or hybrid configurations;
5. plan certificate, key-management and firmware updates;
6. deploy in stages and monitor performance, compatibility and security;
7. retire vulnerable legacy configurations when feasible.

Hybrid approaches may combine a classical and post-quantum mechanism during transition, but the exact security and interoperability properties depend on the protocol and implementation. Larger keys, signatures or ciphertexts can affect bandwidth, storage, latency and embedded devices. PQC is not a single product and does not remove the need for sound key management, authentication or secure implementation.

6. Software Supply-Chain Threats

An organisation’s security depends on code and services it did not write. A supply chain may include operating systems, libraries, containers, package repositories, cloud platforms, firmware, managed service providers and build tools. A compromise at any point can be distributed to many downstream victims.

Threats include:

• malicious or compromised open-source dependencies;
• typosquatting and dependency confusion;
• stolen developer or code-signing credentials;
• tampered build servers and continuous-integration pipelines;
• vulnerable firmware and counterfeit hardware;
• compromised updates from a trusted vendor;
• excessive permissions granted to third-party services;
• outdated components with unpatched vulnerabilities.

Controls should cover the full software-development and procurement lifecycle. They include software bills of materials (SBOMs), dependency pinning, trusted repositories, vulnerability scanning, signed commits and releases, protected build environments, reproducible or verifiable builds where practical, secrets management, supplier due diligence, contractual notification requirements and incident-response coordination.

An SBOM improves visibility but is not proof that software is safe. A listed component may be vulnerable, malicious, incorrectly identified or used in a way not captured by the inventory. Supplier questionnaires are also insufficient without evidence, testing and continuous monitoring.

7. Applied Case: A Financial Services Phishing and AI Incident

The following is an illustrative case, not a report of a specific organisation.

Background

A regional financial institution used email-based invoice approvals and multifactor authentication. Employees completed annual awareness training, and the organisation operated a security-operations centre.

Problem

An attacker used publicly available information to create a convincing message impersonating a senior manager. A voice message, generated with cloned audio, reinforced the request for an urgent change to a supplier’s bank details. The employee followed the normal invoice workflow but did not independently verify the change. A payment was sent before a fraud analyst noticed an unusual destination account.

Analysis

The incident was not caused by one careless individual. The organisation had a process weakness: payment-detail changes could be approved through the same communication channel used to request them. Training discussed suspicious email links but did not adequately address voice deepfakes or urgent payment manipulation. Multifactor authentication protected account access but could not validate the authenticity of the business instruction.

Approach and solution

The organisation introduced an independent callback procedure using supplier contact information already stored in the procurement system. It required dual approval for payment-detail changes, added a reporting route for suspicious voice and video requests, and trained finance staff using realistic but non-punitive exercises. Email authentication was strengthened, high-risk sign-ins were monitored and payment analytics were tuned to identify unusual destinations. An AI assistant was permitted to summarise alerts but not to approve payments or modify supplier records.

Outcome

The revised controls reduced reliance on a single employee’s judgement and created opportunities for detection before funds were transferred. Exercises also showed that staff reported suspicious messages more quickly when the reporting mechanism was prominent and when reports were positively acknowledged.

Learning points

• Multifactor authentication does not prove that a payment instruction is genuine.
• Human awareness must cover modern channels, including synthetic voice and video.
• Independent verification is stronger than visual confidence in a message.
• AI tools require clearly bounded permissions and human accountability.
• Incident analysis should examine process design, not only individual behaviour.

8. Workforce Requirements and Professional Practice

The cyber-security workforce is becoming more interdisciplinary. Technical competence remains essential, but professionals increasingly need communication, risk analysis, privacy awareness, data literacy and the ability to evaluate automated recommendations.

Emerging roles and capabilities include:

• AI security assessment and model-risk management;
• detection engineering and security automation;
• cloud and identity architecture;
• software supply-chain and application-security engineering;
• cryptographic-agility and PQC migration planning;
• digital forensics involving synthetic media;
• privacy, compliance and security governance;
• security education and human-centred design.

Employers should support continuous learning, practical laboratories, incident exercises and cross-functional collaboration. Ethical practice requires respecting privacy, avoiding unnecessary surveillance, documenting decisions and escalating uncertainty. Professionals should be able to explain not only how a control works, but also its assumptions, limitations and effects on people.

Challenges and Limitations

Emerging technologies create uncertainty. AI performance may degrade when attacker behaviour changes, and benchmark results may not represent operational conditions. Human training can become repetitive and ineffective if it is disconnected from real workflows. PQC migration may expose compatibility and performance problems. Supply-chain inventories can be incomplete, especially for transitive dependencies and opaque cloud services. Finally, security controls may impose costs, reduce convenience or create barriers for people with accessibility needs.

These limitations support a risk-based approach. Organisations should prioritise assets and harms, select proportionate controls, test them under realistic conditions and review results over time. Security is not a one-time purchase or a perfect state; it is an ongoing process of reducing likelihood and impact while preserving legitimate functionality.

Best-Practice Checklist

• Make reporting suspicious activity easy, visible and non-punitive.
• Use phishing-resistant authentication for high-value access.
• Verify unusual payment and credential requests independently.
• Treat AI output as assistance requiring validation, not as unquestionable truth.
• Restrict AI tools and agents by identity, data classification and least privilege.
• Test for prompt injection, data leakage, poisoning and unsafe tool use.
• Maintain cryptographic inventories and plan for post-quantum migration.
• Track dependencies, suppliers, build systems and software provenance.
• Measure outcomes and improvement rather than training attendance alone.
• Combine technical controls with usable processes and informed people.

Chapter Summary

Cyber security is socio-technical: people, technology and organisational processes jointly determine risk. Human error is influenced by workload, interface design, authority and process quality, so effective awareness programmes combine role-specific education, usable reporting, technical controls and constructive measurement. Phishing defences should include authentication, email security, independent verification and clear response procedures.

AI can improve detection, triage, threat analysis and secure development, but attackers can use it for scalable deception, automation and synthetic media. Adversarial machine learning includes evasion, poisoning, backdoors, model extraction and membership inference. Generative-AI deployments require controls for prompt injection, data leakage, hallucination, unsafe tool use and overreliance.

Post-quantum cryptography addresses future threats to public-key systems and requires inventory, testing and staged migration. Software supply-chain security requires visibility, trusted development processes, dependency management, supplier governance and continuous monitoring. Future professionals need technical, analytical, ethical and communication skills, together with the ability to evaluate emerging technologies critically.

Key Terms

Human factor: The influence of human behaviour, capability and context on security outcomes.

Social engineering: Manipulation of people to obtain information, access or action.

Phishing: Deceptive communication intended to steal information or induce an unsafe action.

Spear phishing: Targeted phishing directed at a particular person or organisation.

Business email compromise: Fraud involving impersonation or compromise of business communications.

Security awareness: Organised activity that develops secure knowledge, judgement and behaviour.

Machine learning: Methods that learn patterns from data to support prediction or classification.

Generative AI: AI that produces new content such as text, code, images, audio or video.

Hallucination: Plausible but unsupported or incorrect AI output.

Adversarial machine learning: Study of attacks and defences involving machine-learning systems.

Evasion attack: Manipulation of an input to cause an incorrect model decision at use time.

Data poisoning: Corruption of training data to influence a model’s learned behaviour.

Prompt injection: Untrusted instructions that attempt to manipulate a generative-AI system.

Deepfake: Synthetic or manipulated audio, image or video designed to appear authentic.

Post-quantum cryptography: Cryptography intended to resist classical and known quantum attacks.

Harvest now, decrypt later: Collection of encrypted data for possible future decryption.

Software supply chain: The people, code, tools, services and processes used to create and deliver software.

Software bill of materials: An inventory of components contained in a software product.

Cryptographic agility: The capability to change cryptographic algorithms and configurations efficiently.

Least privilege: Giving a user, process or system only the access required for its task.

Review Questions

Short Answer Questions

1. What is the difference between a slip, a mistake and a violation?
2. Name four technical or procedural controls that support phishing resistance.
3. What is an evasion attack in adversarial machine learning?
4. Why is “harvest now, decrypt later” relevant to post-quantum planning?
5. What information does an SBOM provide, and what can it not prove?

Long Answer Questions

1. Explain how a socio-technical view changes the design of security-awareness programmes.
2. Compare defensive and malicious uses of generative AI in cyber security.
3. Describe the main risks of deploying an AI agent that can access organisational tools.
4. Explain a staged strategy for migrating an organisation towards post-quantum cryptography.
5. Discuss how software supply-chain security should extend across development, procurement and incident response.

Analytical / Application Questions

1. A finance employee receives an urgent voice message from an executive requesting a bank-account change. Design a verification process that addresses human, technical and procedural risks.
2. An organisation wants an AI system to automatically disable accounts when it detects anomalous behaviour. Analyse the benefits, false-positive risks, governance requirements and safeguards you would recommend.
3. A critical application depends on hundreds of open-source packages and an external build service. Develop a prioritised supply-chain risk-reduction plan and justify the order of actions.

Multiple Choice Questions with Answers

Question 1

Which measure most directly reduces the risk of an attacker using a stolen password?

A. Annual security training only

B. Phishing-resistant multifactor authentication

C. A larger email mailbox

D. Removing all audit logs

Correct Answer: B. Phishing-resistant multifactor authentication

Question 2

What is the defining purpose of a phishing simulation?

A. To collect employees’ real passwords

B. To punish employees who click links

C. To identify and improve weaknesses in reporting and decision-making

D. To replace technical email controls

Correct Answer: C. To identify and improve weaknesses in reporting and decision-making

Question 3

What is a poisoning attack against a machine-learning system?

A. Changing an input during inference to evade detection

B. Corrupting training data to influence learned behaviour

C. Encrypting a model’s output

D. Replacing a password with a passkey

Correct Answer: B. Corrupting training data to influence learned behaviour

Question 4

Which is an example of indirect prompt injection?

A. A user changes a model’s temperature setting

B. Malicious instructions are embedded in a document retrieved by an AI system

C. A firewall blocks an outgoing connection

D. A developer updates a software library

Correct Answer: B. Malicious instructions are embedded in a document retrieved by an AI system

Question 5

Why is post-quantum migration needed before a large cryptographically capable quantum computer exists?

A. Quantum computers eliminate all symmetric encryption immediately

B. Attackers may collect encrypted data now and decrypt it later

C. PQC removes the need for certificates

D. Classical computers cannot run post-quantum algorithms

Correct Answer: B. Attackers may collect encrypted data now and decrypt it later

Question 6

Which statement about a software bill of materials is correct?

A. It proves every component is vulnerability-free

B. It lists software components and improves visibility of dependencies

C. It replaces code review and secure builds

D. It is only relevant to hardware manufacturers

Correct Answer: B. It lists software components and improves visibility of dependencies

Question 7

Which scenario best illustrates business email compromise?

A. A system automatically patches an operating system

B. An attacker impersonates an executive to request an urgent payment

C. A user changes a screen resolution

D. A developer documents an API

Correct Answer: B. An attacker impersonates an executive to request an urgent payment

Question 8

What is a major limitation of AI-generated security advice?

A. It can never process text

B. It may produce fluent but incorrect information

C. It always requires quantum hardware

D. It cannot be logged

Correct Answer: B. It may produce fluent but incorrect information

Question 9

Which control best addresses an unusual supplier bank-detail change?

A. Independent callback using a trusted contact record

B. Accepting the request because the logo is genuine

C. Disabling all supplier payments permanently

D. Sharing the request publicly for confirmation

Correct Answer: A. Independent callback using a trusted contact record

Question 10

What does least privilege mean in an AI-agent deployment?

A. Giving the agent unrestricted access for convenience

B. Allowing the agent only the data and actions necessary for its approved task

C. Allowing every employee to modify the model

D. Removing all human review

Correct Answer: B. Allowing the agent only the data and actions necessary for its approved task