AI-Enabled Crime, Ethics and Future Developments

Introduction

Artificial intelligence (AI) has become part of the infrastructure through which digital offences are planned, executed and concealed. Generative models can create persuasive text, images, audio and video; predictive systems can identify targets; automation can conduct attacks at scale; and machine-learning tools can alter or fabricate digital evidence. At the same time, law-enforcement agencies use AI to detect fraud, prioritise investigations, analyse large datasets and identify patterns that would be difficult for human investigators to discover.

The central legal challenge is not simply whether AI was used. It is whether the conduct caused legally recognised harm, whether a person or organisation can be held responsible, whether evidence is reliable, and whether state or private responses respect legality, necessity, proportionality, equality and due process. AI changes the speed, scale and plausibility of offending, but it does not remove the importance of human decisions, institutional controls or established principles of cyber law.

This chapter examines deepfakes, synthetic identities, automated phishing, adversarial AI, algorithmic evidence and AI-assisted policing. It also considers ethical risks, governance requirements and likely legal developments in emerging digital environments.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain how AI enables contemporary forms of cybercrime and digital offending.
2. Distinguish deepfakes, synthetic identities, automated phishing and adversarial attacks.
3. Assess the evidential value and limitations of AI-generated or AI-analysed material.
4. Apply the principles of legality, accountability, fairness, privacy and proportionality to AI systems.
5. Evaluate AI-assisted policing and identify appropriate safeguards.
6. Analyse a practical incident involving AI-enabled fraud and recommend lawful responses.
7. Identify likely directions of legal and regulatory development.

1. AI as a Capability Multiplier

AI is best understood as a capability multiplier rather than an independent criminal actor. A criminal may use an AI system to produce messages, translate content, imitate a voice, search public information, write malicious code or select targets. The system increases speed and scale, but responsibility usually remains connected to the people and organisations that design, deploy, direct or negligently control it.

AI-enabled offending commonly involves five capabilities:

AI can also lower the skill barrier. A person with limited technical knowledge may use commercial interfaces to conduct activities that previously required specialist expertise. This democratisation of capability affects threat assessment: the relevant question is not only what a highly skilled attacker can do, but what can be done cheaply and repeatedly by ordinary users with access to public tools.

2. Deepfakes and Synthetic Media

A deepfake is digitally generated or manipulated audio, image or video that appears to represent a real person, event or statement. Modern generative systems can create synthetic media through methods such as generative adversarial networks, diffusion models and speech-synthesis systems. The legal significance depends on context. A fictional film scene is not equivalent to a fabricated video used to obtain money, damage reputation or influence an election.

2.1 Common forms of misuse

Deepfakes may be used for:

• executive impersonation and fraudulent payment instructions;
• non-consensual intimate imagery;
• false evidence of misconduct or criminality;
• political manipulation and fabricated public statements;
• extortion and blackmail;
• evasion of identity verification;
• reputational attacks against businesses or individuals.

Deepfake harm is often intensified by the “liar’s dividend”: once synthetic media becomes common, genuine recordings may be dismissed as fake. This weakens public confidence in visual and auditory evidence and makes rapid verification essential.

2.2 Legal analysis

Potential offences include fraud, deception, identity-related offences, harassment, extortion, privacy violations, malicious communications, electoral offences and offences involving intimate images. Civil liability may arise through defamation, misuse of private information, breach of confidence, data-protection violations or interference with contractual relations. The applicable offence depends on jurisdiction, intent, harm, publication and the victim’s status.

A deepfake is not automatically unlawful merely because it is artificial. Important questions include:

1. Was the material presented as authentic?
2. Was there an intention to deceive or cause harm?
3. Was consent obtained from the depicted person?
4. Was the content defamatory, intimate, threatening or fraudulent?
5. Who created, commissioned, distributed or knowingly amplified it?
6. Did a platform or intermediary have notice and a legal duty to respond?

2.3 Detection and authentication

Detection tools may examine visual artefacts, audio inconsistencies, compression patterns, file metadata, physiological signals or generation fingerprints. These methods are imperfect. Re-encoding can remove metadata, detection models may fail on unfamiliar generators, and an ordinary recording can be falsely classified as synthetic.

Accordingly, authentication should be layered. Investigators should preserve the original file, record acquisition circumstances, obtain platform records, compare independent recordings, examine device and account logs, and document every transformation. A detector’s output should normally be treated as an investigative lead or probabilistic indicator, not conclusive proof.

3. Synthetic Identities and Identity Fraud

A synthetic identity combines genuine and fabricated information to create a person who does not fully exist in the claimed form. For example, an offender may combine a real national identification number with a false name, address and employment history. AI improves this technique by generating plausible biographies, profile photographs, documents, social activity and conversational behaviour.

Synthetic identities differ from simple impersonation. In impersonation, the offender commonly pretends to be a particular real person. In synthetic identity fraud, the identity may be assembled from multiple sources or invented entirely. Both can overlap in practice.

3.1 Criminal process

A typical scheme may involve:

1. collecting personal information from breaches, public sources or social engineering;
2. constructing a consistent identity profile;
3. establishing a digital history through accounts and interactions;
4. passing weak customer-identification checks;
5. obtaining credit, benefits, services or access;
6. building trust before committing a larger fraud;
7. abandoning the identity or transferring control to another actor.

AI-generated faces and voices can defeat poorly designed know-your-customer procedures. However, biometric matching is not equivalent to identity verification. A system may correctly determine that a face matches an enrolled image while failing to establish whether the person is authorised to use that identity.

3.2 Controls

Effective controls combine document verification, device intelligence, behavioural analysis, liveness testing, risk-based authentication, transaction monitoring and human review. Organisations should avoid relying on a single biometric or automated score. Controls must also comply with privacy, data-protection and equality requirements, especially where biometric data is processed.

4. Automated Phishing and Social Engineering

Phishing is the use of deceptive communications to induce a person to reveal information, transfer funds, install software or take another harmful action. AI enhances phishing by producing grammatically accurate messages, translating them into local languages, imitating corporate style and generating personalised content from publicly available information.

Automated phishing may include:

• email campaigns generated and sent at scale;
• AI chatbots that maintain conversations with victims;
• voice-cloning calls from supposed relatives or managers;
• automated reconnaissance of professional and social profiles;
• adaptive scams that change after observing a victim’s responses.

The traditional distinction between “mass phishing” and “spear phishing” is therefore becoming less clear. Automation can provide mass campaigns with the personalisation formerly associated with highly targeted attacks.

Professional response

Organisations should combine technical and human measures. Email authentication protocols, secure payment procedures, multifactor authentication, domain monitoring, staff training and out-of-band confirmation are complementary controls. A payment request that is unusual, urgent or confidential should be independently verified using a trusted channel, not by replying to the message that made the request.

From a legal perspective, the analysis may involve unauthorised access, fraud, identity offences, malicious communications, data-protection violations and conspiracy or participation offences. Evidence should preserve message headers, domain-registration information, authentication records, payment trails, account logs and relevant communications.

5. Adversarial AI and Attacks on Machine-Learning Systems

Adversarial AI concerns deliberate attempts to manipulate, deceive or exploit machine-learning systems. An attacker may alter an input so that a model produces an incorrect output, poison training data, extract sensitive information, evade detection or cause a model to disclose protected material.

Important categories include:

• Evasion attacks: modified inputs avoid detection or classification.
• Data poisoning: malicious training examples influence later behaviour.
• Model extraction: repeated queries are used to approximate a proprietary model.
• Membership inference: an attacker tests whether particular data appeared in training.
• Prompt injection: instructions embedded in data or documents manipulate a generative model.
• Model inversion: outputs are used to infer features of training data.

The legal consequences vary. Conduct may constitute unauthorised access, interference with computer systems, theft of trade secrets, breach of confidence, data misuse or fraud. Some research activities may be legitimate security testing if properly authorised. Authorisation, scope, intent and harm are therefore critical.

Security and governance response

Organisations should maintain an inventory of AI systems, classify their risks, restrict access, test models against realistic attacks, monitor input and output behaviour, validate training data and retain meaningful logs. High-impact decisions should have a human review route and a mechanism for correction. Security testing should be documented through written authorisation, defined boundaries and responsible disclosure procedures.

6. Algorithmic Evidence

Algorithmic evidence is information produced, selected or interpreted by an automated or AI-based system. Examples include a facial-recognition match, a fraud-risk score, a malware classification, an automated transcription or a system-generated explanation of suspicious behaviour.

The existence of an algorithmic output does not answer the legal question of admissibility or weight. Courts commonly consider relevance, authenticity, reliability, procedural fairness, expert foundation and statutory rules. The precise test differs across legal systems.

6.1 Evidence lifecycle

A reliable evidential process should address the following stages:

1. Collection: How was the data acquired, and was acquisition lawful?
2. Integrity: Was the original preserved, and can alteration be detected?
3. Processing: What software, version and settings were used?
4. Validation: Was the system tested for accuracy and known failure modes?
5. Interpretation: What does the output actually show, and what does it not show?
6. Disclosure: Can the opposing party examine relevant data, methods and limitations?
7. Presentation: Is the explanation understandable without overstating certainty?

A hash value can help demonstrate that a digital file has remained unchanged since a particular point in the chain of custody. It does not prove that the file was truthful when originally created, nor does it prove that the device or account owner caused the file to exist.

6.2 Probabilistic outputs

Many AI systems produce probabilities, rankings or confidence scores. These are not automatically legal probabilities of guilt. A model’s confidence may reflect statistical similarity under test conditions rather than the probability that a particular person committed an offence.

Investigators should distinguish:

• the model output, such as a similarity score;
• the error rate, including false positives and false negatives;
• the base-rate context, meaning how common the event is;
• the independent evidence, such as access logs or witness testimony;
• the decision threshold, which determines how the output is used.

A high score may justify further investigation but should not, without more, determine arrest, conviction or punishment.

7. AI-Assisted Policing and Criminal Justice

Police and justice agencies may use AI for image comparison, predictive deployment, link analysis, transcription, digital triage, risk assessment, online threat detection and prioritisation of investigative leads. These applications can reduce administrative burdens and help process large volumes of material. They also create significant risks.

7.1 Predictive policing

Predictive policing systems may forecast locations, times or persons associated with increased risk. Historical data, however, reflects prior enforcement patterns, reporting practices and institutional bias. If police have historically concentrated in a neighbourhood, that neighbourhood may generate more recorded incidents, which can then appear to justify further concentration. This creates a feedback loop rather than an objective discovery of criminality.

A risk forecast is not proof of individual wrongdoing. Ethical and legal safeguards should prohibit treating a prediction as a substitute for reasonable suspicion or legally sufficient evidence.

7.2 Facial recognition and biometric systems

Facial recognition may be used for identification, verification or investigation. These are different functions and should not be treated as interchangeable. A one-to-one verification question asks whether a person matches a claimed identity. A one-to-many identification search asks whether a person resembles anyone in a database. The latter creates greater risks of false matches, indiscriminate monitoring and chilling effects on lawful activity.

Deployment should be governed by a clear legal basis, defined purpose, necessity assessment, accuracy testing across relevant demographic groups, retention limits, access controls, auditability and a process for challenge and correction.

7.3 Due process and human oversight

Human involvement is not meaningful merely because a person presses an approval button. A human reviewer must have sufficient information, authority, time and independence to question the system. Agencies should record the system used, the input data, the output, the decision-maker, the reasons for action and any later correction.

8. Ethical and Legal Principles

Legality and foreseeability

People should be able to know in advance what conduct is prohibited. New technology should not be criminalised through vague interpretation alone. Legislatures may need to update offences relating to impersonation, synthetic media, non-consensual intimate imagery, automated attacks and misuse of biometric information, while preserving clear elements of liability.

Accountability

Accountability should follow control and contribution. Relevant actors may include the offender, a contracting organisation, a model provider, a platform, an employer and a public authority. Liability should not be imposed automatically on a developer for every misuse, but foreseeable risks, warnings, access controls, monitoring and response capacity are relevant to negligence and regulatory responsibility.

Proportionality

A response should be suitable for the legitimate objective, necessary in the circumstances and balanced against rights and harms. Blocking an entire service because a small number of users misuse it may be disproportionate. Conversely, allowing an easily foreseeable deepfake abuse to continue after notice may be inadequate.

Fairness and non-discrimination

AI systems can reproduce historical inequalities or create new ones through unrepresentative data, proxy variables and unequal error rates. Fairness is not a single technical property. It may require testing performance across groups, examining the consequences of errors, consulting affected communities and providing an effective appeal process.

Privacy and data minimisation

AI systems often require large datasets, including personal and biometric information. Data should be collected for a defined purpose, retained no longer than necessary, protected against misuse and processed under a lawful basis. Public availability does not automatically eliminate privacy interests.

Transparency and explainability

Transparency can mean disclosure of the system’s purpose, data categories, performance, limitations and governance arrangements. It does not always require publication of source code. In high-impact settings, however, an affected person should receive enough information to challenge a decision effectively.

9. Applied Case: The Fabricated Executive Instruction

The following is an illustrative fictional case.

Background

A medium-sized company receives a video call from a person who appears to be its finance director. The voice, face and mannerisms are convincing. The person instructs the finance manager to transfer funds to a new supplier account and insists that the transaction is confidential because of an acquisition. The manager complies. The account is controlled by criminals.

Problem

The company must determine whether the event was an ordinary business-email compromise, a deepfake incident, an insider offence or a combination. It must preserve evidence, attempt recovery, notify relevant authorities and decide whether any employee or service provider was negligent.

Analysis

Initial indicators include unusual urgency, secrecy, a new payment destination and a request to bypass normal approval controls. The video’s apparent authenticity is not decisive. Investigators should preserve the recording, call metadata, email and messaging logs, payment instructions, account records, device data and relevant access logs. They should obtain independent confirmation from the genuine executive and examine whether credentials were compromised.

The criminal analysis may include fraud, impersonation, unauthorised access, money laundering and conspiracy. The civil and regulatory analysis may involve contractual duties, data protection, payment-services rules, insurance conditions and organisational security obligations. Liability cannot be decided merely because an employee trusted the video; the organisation’s approval procedures and the reasonableness of the employee’s conduct must be assessed.

Approach and solution

The company should immediately contact the bank and request recall or freezing of funds, preserve evidence under a documented chain of custody, isolate compromised accounts, notify insurers and report the incident where required. It should introduce a rule that high-value or unusual transfers require independent confirmation through a pre-registered channel and approval by two authorised persons. Video or voice alone must not authorise payment.

Outcome and learning points

The funds are partly recovered because the bank is notified quickly. The investigation finds that publicly available executive interviews were used to train a voice and video imitation, while the criminal group also obtained internal supplier information through a compromised mailbox.

The case demonstrates four principles:

1. Synthetic media often operates with conventional credential theft and social engineering.
2. Authentication of content is different from authentication of authority.
3. Procedural controls remain essential even when technical detection fails.
4. Fast preservation and independent verification are both legal and operational priorities.

10. Likely Legal and Regulatory Developments

Legal development is likely to occur through a combination of new legislation, regulatory standards, judicial interpretation and institutional guidance. Important directions include:

• specific offences for malicious creation or distribution of intimate deepfakes;
• duties concerning notice, removal, provenance and reporting of synthetic media;
• stronger rules for biometric identification and high-risk automated decision-making;
• mandatory impact assessments, risk management and incident reporting;
• disclosure obligations for AI-generated evidence used in proceedings;
• provenance or authenticity standards for media and official communications;
• enhanced duties for platforms, model providers and deployers;
• rules governing cross-border access to data and AI-enabled investigations;
• procurement requirements requiring public authorities to demonstrate accuracy, necessity and auditability.

A continuing difficulty is jurisdiction. An offender may operate in one country, use infrastructure in a second, target a victim in a third and place stolen funds in a fourth. International cooperation, electronic-evidence agreements, mutual legal assistance and compatible preservation procedures will therefore remain important.

Lawmakers must also avoid overbroad rules that suppress satire, journalism, artistic expression, legitimate security research or political criticism. A risk-based approach is generally more adaptable than attempting to prohibit every new technical method. Regulation should focus on conduct, harm, context, foreseeability and safeguards.

11. Best-Practice Framework

Organisations handling AI should adopt the following minimum programme:

1. Inventory: Identify AI systems, datasets, vendors and uses.
2. Classify risk: Give greater scrutiny to systems affecting identity, employment, finance, liberty, safety or access to essential services.
3. Control access: Apply authentication, authorisation, logging and least privilege.
4. Test before deployment: Assess accuracy, bias, security, robustness and misuse scenarios.
5. Monitor continuously: Detect drift, abuse, unusual inputs and unexplained performance changes.
6. Preserve evidence: Maintain reliable logs, version records and chain-of-custody procedures.
7. Provide human review: Ensure meaningful challenge, correction and escalation.
8. Train personnel: Teach staff to recognise synthetic media, automated phishing and authority fraud.
9. Plan incidents: Define reporting, containment, notification and recovery responsibilities.
10. Review vendors: Require contractual commitments on security, privacy, audit access, incident notification and data use.

The objective is not to eliminate all risk, which is unrealistic, but to make risk visible, controlled and reviewable.

Challenges and Limitations

AI detection is an arms race: generators improve as detectors learn. Technical accuracy may vary across languages, devices, demographic groups and content types. Explainability methods can simplify a complex model without genuinely revealing its reasoning. Human review can reduce error but can also reproduce institutional bias or become a rubber stamp. Regulatory requirements may conflict across jurisdictions, creating compliance uncertainty. Finally, excessive reliance on automated tools may produce automation bias, in which professionals accept a machine recommendation despite contrary evidence.

For these reasons, AI should be treated as one source of information within a legally governed process. Good governance requires technical safeguards, institutional competence, documented reasoning and respect for affected persons.

Chapter Summary

AI enables crime by multiplying the speed, scale, plausibility and adaptability of deception and attack. Deepfakes can support fraud, extortion, reputational harm and political manipulation, while synthetic identities exploit weaknesses in identity and customer-verification systems. Automated phishing makes personalised social engineering inexpensive and scalable. Adversarial AI targets the models themselves through evasion, poisoning, extraction and prompt manipulation.

Algorithmic evidence must be assessed through its collection, integrity, processing, validation, interpretation, disclosure and presentation. A model score is not automatically proof, and human oversight must be meaningful rather than ceremonial. AI-assisted policing can improve investigative capacity but raises serious concerns about privacy, bias, false positives, feedback loops and due process.

The central governance principles are legality, accountability, proportionality, fairness, privacy, transparency and contestability. Future legal developments are likely to address synthetic media, biometric systems, high-risk automated decisions, platform duties, AI evidence and cross-border cooperation. Effective practice combines technical security with independent verification, human judgement, reliable records and clear institutional responsibility.

Key Terms

Deepfake: Synthetic or manipulated audio, image or video that appears to depict a real person or event.

Synthetic media: Digitally generated or altered content, including text, images, audio and video.

Synthetic identity: An identity assembled from genuine and fabricated personal information.

Automated phishing: AI-supported deceptive communication conducted at scale or adapted automatically.

Social engineering: Manipulation of people to obtain information, access, money or action.

Adversarial AI: Deliberate attempts to manipulate, evade or exploit machine-learning systems.

Evasion attack: Modification of an input to cause a model to misclassify it.

Data poisoning: Insertion of malicious training data to influence model behaviour.

Prompt injection: Instructions embedded in input data to manipulate a generative model.

Algorithmic evidence: Information produced or interpreted by an automated system and used in an investigation or proceeding.

False positive: An incorrect result indicating that a condition or match exists.

False negative: An incorrect result failing to identify a condition or match.

Automation bias: Undue reliance on an automated recommendation.

Predictive policing: Use of analytical systems to forecast crime-related risks or priorities.

Biometric data: Data relating to physical or behavioural characteristics used for identification or verification.

Proportionality: The requirement that a measure be suitable, necessary and balanced against its effects.

Chain of custody: Documented history of the handling, preservation and transfer of evidence.

Data poisoning: Manipulation of training data to corrupt a machine-learning system.

Human oversight: Effective human capacity to understand, question, override or correct an automated process.

Liar’s dividend: The ability to dismiss genuine material as fake because synthetic media is widely expected.

Review Questions

Short Answer Questions

1. What distinguishes a synthetic identity from simple impersonation?
2. Why is a deepfake detector’s result not normally conclusive evidence?
3. Name three types of adversarial attack against AI systems.
4. What is automation bias?
5. State two safeguards for AI-assisted policing.

Long Answer Questions

1. Explain how AI changes the scale and effectiveness of phishing and social engineering.
2. Discuss the evidential challenges created by AI-generated or AI-analysed material.
3. Analyse the relationship between predictive policing, historical data and institutional bias.
4. Explain how legality, accountability and proportionality should guide AI regulation.
5. Describe an organisational governance framework for managing high-risk AI systems.

Analytical / Application Questions

1. A company receives a convincing voice message from its chief executive requesting an urgent transfer. What immediate evidential, technical and legal steps should it take?
2. A facial-recognition system produces a high-confidence match, but the person denies involvement and the system has unequal error rates across demographic groups. How should investigators and a court treat the result?
3. A platform is notified that a fabricated intimate video is being distributed. Assess the responsibilities of the creator, distributor and platform, taking account of consent, harm, notice and proportionality.

Multiple Choice Questions with Answers

Question 1

Which feature most clearly distinguishes a synthetic identity from ordinary impersonation?

A. It always uses stolen passwords

B. It combines genuine and fabricated information to construct an identity

C. It can only be used for social-media offences

D. It requires a deepfake video

Correct Answer: B. It combines genuine and fabricated information to construct an identity

Question 2

Why should an AI-generated confidence score not automatically be treated as proof of guilt?

A. AI systems never produce numerical outputs

B. A score may reflect statistical similarity rather than legal responsibility

C. Courts cannot consider digital evidence

D. Confidence scores are always deliberately falsified

Correct Answer: B. A score may reflect statistical similarity rather than legal responsibility

Question 3

Which is an example of a data-poisoning attack?

A. Changing training examples so that a model later behaves incorrectly

B. Using two-factor authentication to protect a model

C. Preserving a file with a hash value

D. Reviewing a model’s output by a human

Correct Answer: A. Changing training examples so that a model later behaves incorrectly

Question 4

What is the principal danger of treating a predictive-policing forecast as proof of individual wrongdoing?

A. It makes evidence impossible to store

B. It substitutes a statistical prediction for legally sufficient evidence

C. It prevents all future software updates

D. It guarantees that no offence occurred

Correct Answer: B. It substitutes a statistical prediction for legally sufficient evidence

Question 5

Which control is most appropriate for an unusual high-value payment request received by video call?

A. Relying solely on the apparent realism of the video

B. Replying to the same message for confirmation

C. Independent confirmation through a trusted pre-established channel

D. Disabling all payment systems permanently

Correct Answer: C. Independent confirmation through a trusted pre-established channel

Question 6

What does a hash value primarily help demonstrate in digital evidence?

A. That the file was truthful when created

B. That the file has remained unchanged since hashing

C. That the suspect created the file

D. That an AI detector is unbiased

Correct Answer: B. That the file has remained unchanged since hashing

Question 7

Which principle requires an AI measure to be suitable, necessary and balanced against its effects?

A. Proportionality

B. Automation bias

C. Model extraction

D. Data minimisation

Correct Answer: A. Proportionality

Question 8

Which statement best describes prompt injection?

A. Physical damage to a computer server

B. Instructions embedded in input data that manipulate a generative model

C. A method for encrypting biometric data

D. A court procedure for authenticating witnesses

Correct Answer: B. Instructions embedded in input data that manipulate a generative model

Question 9

Why can historical data create a feedback loop in predictive policing?

A. Historical data is never recorded

B. Past enforcement patterns can influence future predictions and enforcement

C. Predictions automatically delete police records

D. Police systems cannot use geographic information

Correct Answer: B. Past enforcement patterns can influence future predictions and enforcement

Question 10

Which statement best reflects meaningful human oversight?

A. A person approves every output without seeing the basis for it

B. A human has the information, authority and opportunity to question or override the system

C. The system is described as intelligent in a policy document

D. The organisation removes all appeal procedures

Correct Answer: B. A human has the information, authority and opportunity to question or override the system