Artificial Intelligence, Profiling and Automated Decisions

Introduction and Context

Artificial intelligence is increasingly used to classify people, predict behaviour, recommend opportunities, detect fraud, assess risk and automate administrative decisions. These uses can improve speed and consistency, but they also create significant privacy risks. An AI system may process personal data at several stages: collection, dataset construction, model training, deployment, inference, sharing and retention.

Privacy law therefore concerns more than whether an organisation has obtained data lawfully. It also concerns whether the data are appropriate for the purpose, whether individuals can understand and challenge the system, whether the system produces discriminatory outcomes, and whether meaningful human judgment remains available.

This chapter focuses on the relationship between data protection law and AI. It uses the GDPR as a central reference because its principles and rights have strongly influenced global practice, while also noting the EU Artificial Intelligence Act and other regulatory approaches. Legal obligations depend on jurisdiction, sector, role and system risk. This chapter is educational and does not replace jurisdiction-specific legal advice.

Learning Objectives

By the end of this chapter, a learner should be able to:

1. Explain how privacy risks arise in AI datasets, training and inference.
2. Distinguish profiling from automated decision-making.
3. Apply data-protection principles to AI projects.
4. Identify requirements concerning transparency, explainability, bias and human oversight.
5. Evaluate a basic AI use case and recommend proportionate safeguards.
6. Describe important contemporary developments in AI regulation.

1. Essential Concepts

Artificial intelligence, machine learning and inference

Artificial intelligence is a broad term for computational systems that perform tasks commonly associated with human intelligence, such as prediction, classification, language processing or image recognition. Machine learning is a method in which a system identifies patterns from data rather than following only hand-written rules.

A trained model receives an input and produces an output. The output may be a probability, category, ranking, recommendation or generated response. The process of applying a trained model to new data is inference. Personal data may be involved in both the input and the output. A model can therefore create privacy risks even when it does not store a conventional database of individual records.

Personal data and special categories

Under the GDPR, personal data means information relating to an identified or identifiable natural person. Identifiability may arise directly, through a name or identifier, or indirectly, through combinations such as location, device information and behavioural history. Pseudonymised data remain personal data where re-identification remains reasonably possible. Properly anonymised data fall outside the GDPR, although anonymisation is difficult and must be assessed against realistic means of re-identification.

Special-category data include information revealing racial or ethnic origin, political opinions, religious or philosophical beliefs, trade-union membership, genetic data, biometric data used for uniquely identifying a person, health data, and information concerning sex life or sexual orientation. Processing normally requires both a general lawful basis and an additional condition under data-protection law.

Profiling

Profiling is automated processing of personal data used to evaluate or predict aspects of a person, including their performance at work, economic situation, health, preferences, reliability, behaviour, location or movements. Examples include assigning a credit-risk score, predicting insurance claims, targeting advertisements or ranking job applicants.

Profiling is not necessarily prohibited. Its legality depends on purpose, lawful basis, fairness, transparency, necessity, proportionality and applicable rights. A profile may be inferred rather than directly collected. An individual who has never stated that they are financially vulnerable may nevertheless be assigned that characteristic by a model.

Automated decision-making

Automated decision-making occurs when a decision is made by technological means without meaningful human involvement. The decision may concern access to employment, credit, education, insurance, public services or online accounts.

A useful distinction is between:

• automated assistance, where a human genuinely evaluates the recommendation;
• automated decision-making, where the system determines the outcome; and
• decision support with nominal human involvement, where a human formally approves an output but merely rubber-stamps it.

The third category is legally and ethically important. Human presence alone does not establish meaningful human oversight.

2. The AI Data Lifecycle and Privacy Risk

Privacy governance should follow the entire lifecycle rather than focusing only on deployment.

Figure: Privacy controls should cover the complete AI lifecycle.

Collection and dataset construction

Data may be collected directly, purchased from a broker, scraped from public websites, obtained from partners or generated by users. Public availability does not automatically make processing lawful or fair. The organisation must still consider purpose, expectations, legal basis, transparency and data minimisation.

Datasets may contain inaccurate labels, historic discrimination, duplicate records, hidden sensitive attributes or data collected for an incompatible purpose. A dataset can also be unrepresentative. If a facial-recognition dataset contains few images of darker-skinned people, performance may differ materially between groups.

Training and memorisation

During training, a model may learn relationships that reflect social prejudice or unlawful proxies. For example, postcode may correlate with ethnicity or income; school attended may correlate with protected characteristics. Removing a sensitive field does not necessarily remove discrimination because other features can act as proxies.

Large models may memorise unusual or repeated information, including names, contact details or confidential text. Memorisation is not equivalent to proof that training data can always be extracted, but it creates a risk requiring testing, access controls and appropriate contractual safeguards.

Inference and function creep

Inference may reveal new information. A system can infer health status, political interest or emotional state from seemingly ordinary data. The inferred result may be personal data even if the organisation never collected the underlying fact directly.

Function creep occurs when data collected for one purpose are later used for another without adequate compatibility analysis or a new legal basis. A customer-support transcript, for example, may be repurposed to train a general model without sufficient notice or controls.

Retention, sharing and security

AI projects often create multiple copies: raw data, cleaned datasets, feature stores, logs, prompts, outputs, model checkpoints and backups. Retention schedules must address all of them. Security should include role-based access, encryption where appropriate, supplier controls, testing, logging and procedures for deletion or correction.

3. Applying Core Data-Protection Principles

The GDPR principles provide a practical governance framework.

Lawfulness and accountability

An organisation should identify its role and lawful basis before development. Consent is not a universal solution: it must be freely given, specific, informed and withdrawable, and may be unsuitable where there is an imbalance of power. Legitimate interests require a documented balancing assessment. Public authorities and employers may face additional constraints.

A data-protection impact assessment is generally appropriate, and may be required, where processing is likely to result in a high risk to individuals, including systematic and extensive profiling or large-scale use of sensitive data. The assessment should describe processing, assess necessity and proportionality, identify risks and record mitigating measures.

Data minimisation and privacy-enhancing methods

Minimisation means more than deleting names. It asks whether each feature, record, granularity level and retention period is necessary. Possible techniques include aggregation, pseudonymisation, access separation, synthetic data, federated learning and differential privacy. None is automatically risk-free. Synthetic data can reproduce source-data bias; federated learning can still leak information through updates; and differential privacy involves a privacy-utility trade-off.

A common formal expression for differential privacy is:

Here, M is a randomised mechanism, D and D′ are neighbouring datasets differing in one person’s record, S is a set of possible outputs, and ε is a privacy-loss parameter. Smaller ε generally provides stronger privacy, subject to the mechanism and composition of repeated releases. The expression applies only when the mechanism has been correctly designed and the privacy budget is managed; it does not guarantee that every use of the resulting data is lawful.

4. Transparency and Explainability

Transparency is a legal and governance requirement, not merely a technical feature. Individuals should usually be informed about the purposes of processing, categories of data, relevant logic or meaningful information about the logic, consequences and available rights. Information should be intelligible and proportionate to the audience.

Explainability has several meanings:

• Global explanation: how the system generally works and which variables influence outputs.
• Local explanation: why a particular person received a particular result.
• Process explanation: who supplied data, what checks occurred and how to challenge the outcome.
• Counterfactual explanation: what relevant change might have produced a different result, where such an explanation is reliable and legally appropriate.

An explanation must not be misleading. A post-hoc explanation that merely lists correlations may not reflect the model’s actual operation. Nor should trade secrecy be used as a blanket reason to deny legally required information, although disclosure must be balanced with security, intellectual property and the rights of others.

5. Bias, Fairness and Accuracy

Bias can enter through historical decisions, sampling, measurement, labels, missing data, feature selection, deployment context and feedback loops. A model can be statistically accurate overall while producing unacceptable errors for a particular group.

Fairness is not a single technical property. Relevant questions include whether error rates differ, whether access is equitable, whether the target variable is a valid measure, and whether the use itself is justified. Some fairness criteria cannot all be satisfied simultaneously when group base rates differ. Consequently, technical metrics must be interpreted alongside legal, social and organisational analysis.

Useful checks include subgroup performance, false-positive and false-negative rates, calibration, data coverage, error analysis and human impact assessment. Organisations should document which groups were considered, which metric was selected, why it is appropriate and what limitations remain.

Accuracy also has legal significance. Incorrect source data can lead to unfair decisions, while a statistically accurate model may still be unlawful if the purpose lacks a legal basis or the processing is disproportionate. Individuals should have practical routes to correct relevant data and contest outcomes.

6. Human Oversight and Individual Rights

Under GDPR Article 22, individuals have protections concerning decisions based solely on automated processing, including profiling, that produce legal effects or similarly significant effects, subject to specified conditions and safeguards. The precise scope and exceptions depend on the applicable law and interpretation.

Where automated decision-making is permitted, safeguards may include information about the decision, an opportunity to obtain human intervention, an opportunity to express a view, and a right to contest the decision. Human intervention must be competent, independent enough to question the system, adequately informed and empowered to change the outcome.

Other relevant rights may include access, rectification, erasure, restriction, objection and data portability, subject to conditions and exceptions. Rights requests involving AI can be difficult because data may exist in training records, vector stores, logs or model parameters. A controller should therefore design systems with traceability and rights-response procedures from the beginning.

7. Emerging Regulation and Governance

The EU AI Act adopts a risk-based structure. It prohibits certain unacceptable-risk practices, imposes obligations on high-risk systems, and creates transparency duties for specified systems such as certain chatbots and synthetic or manipulated content. Obligations vary by provider, deployer, importer and distributor, and by the system’s classification. Organisations must examine the current legal text, transition dates and sector-specific rules rather than relying on broad summaries.

The Act does not replace data-protection law. A system may comply with one regime and breach another. Other relevant instruments include the Council of Europe framework on AI and human rights, national automated-decision laws, equality legislation, consumer-protection law, employment law, sectoral rules and emerging governance standards.

Effective governance commonly includes:

1. an inventory of AI systems and use cases;
2. risk classification and impact assessment;
3. clear allocation of controller, processor, provider and deployer responsibilities;
4. dataset provenance and documentation;
5. testing for accuracy, robustness, security and discriminatory impact;
6. human-oversight procedures;
7. incident reporting and change management;
8. monitoring after deployment; and
9. retirement and deletion controls.

Applied Case: Automated Recruitment Screening

Background

This illustrative case concerns Northbridge Services, a medium-sized company that uses an AI tool to rank job applications. The vendor claims that the system predicts which applicants are most likely to perform well by analysing CV text and application answers.

Problem

The company discovers that applicants from certain universities and applicants with career breaks are ranked lower. The vendor had trained the system on ten years of historical recruitment data. Those historical decisions were made by managers and were not assessed for discrimination. The company also cannot clearly explain individual rankings.

Analysis

The historical dataset is not a neutral measure of merit. It reflects previous hiring preferences and may encode indirect discrimination. University name, postcode, language style and employment gaps may operate as proxies for protected or socio-economic characteristics. The company is also processing personal data for a new purpose and must assess lawful basis, transparency, necessity, retention and supplier responsibilities.

The system may constitute profiling. If it automatically rejects candidates or effectively determines who reaches interview, the safeguards for significant automated decisions may apply. A manager who approves every ranking without reviewing evidence is unlikely to provide meaningful human involvement.

Approach and solution

Northbridge should pause automatic rejection and conduct a data-protection and equality impact assessment. It should document the purpose and assess whether AI ranking is necessary at all. The company should remove unnecessary features, test performance and error rates across relevant groups, review labels and historical outcomes, and require the vendor to provide sufficient documentation and audit access.

A trained recruiter should review applications using the model only as a limited aid, record reasons for departures from the recommendation, and be empowered to override it. Applicants should receive clear information about the processing and available challenge mechanisms. Data retention should be limited, and unsuccessful applicants’ information should not be retained indefinitely for unspecified future training.

Outcome

In this illustrative case, testing shows that the model’s apparent overall accuracy was acceptable, but the interview-selection rate for applicants with career gaps was substantially lower. The company withdraws the feature associated with career-gap penalties, introduces structured human review and commissions periodic independent audits. It also decides that some roles do not justify automated ranking.

Learning points

The case demonstrates that technical performance does not establish fairness or legality. Historical data may encode institutional choices; human oversight must be substantive; vendor procurement is part of privacy governance; and a less automated process may be the most proportionate solution.

Challenges and Limitations

AI governance faces several practical limitations. First, complex models can be difficult to explain without oversimplification. Second, fairness metrics depend on disputed choices about groups, labels and acceptable error. Third, privacy-preserving techniques may reduce utility or introduce new risks. Fourth, supply chains make responsibility difficult: a deployer may lack access to training data or model documentation. Fifth, models and data change over time, so a one-time assessment is inadequate.

There are also deeper concerns. Profiling can shape opportunities rather than merely describe them, producing feedback loops in which a prediction influences the very behaviour later used to validate it. Excessive transparency may facilitate gaming or expose security-sensitive information, while insufficient transparency undermines accountability. Governance must therefore be proportionate, evidence-based and revisited when systems, purposes or populations change.

Best-Practice Checklist

Before deployment, an organisation should be able to answer:

• What precise decision or task is the system supporting?
• Is the purpose lawful, necessary and proportionate?
• What data are used, where did they come from and how long will they be retained?
• Are special-category data or sensitive proxies involved?
• Who is affected, including non-users and vulnerable groups?
• What errors and discriminatory impacts have been tested?
• Can an individual obtain a meaningful explanation and challenge the outcome?
• Is human oversight real, skilled and empowered?
• Can data be corrected, deleted or isolated when required?
• What contracts, audits, monitoring and incident procedures apply?

Chapter Summary

AI creates privacy issues throughout the data lifecycle, from collection and dataset construction to training, inference, retention and deletion. Profiling evaluates or predicts characteristics of individuals, while automated decision-making uses technological means without meaningful human involvement. These activities are not automatically unlawful, but they require careful attention to lawful basis, fairness, transparency, purpose limitation, minimisation, accuracy, security and accountability.

Removing names does not necessarily anonymise data, and removing sensitive attributes does not eliminate discriminatory proxies. Explainability should address both system logic and the practical process for challenging an outcome. Human oversight must involve competent, informed and empowered decision-makers. Impact assessments, documentation, subgroup testing, supplier controls and ongoing monitoring are central safeguards.

The GDPR, equality law, sectoral rules and emerging risk-based AI regulation operate together. Responsible practice therefore combines legal analysis, technical testing, organisational governance and respect for human dignity.

Key Terms

Artificial intelligence: Computational systems performing tasks associated with human intelligence.

Automated decision-making: A decision made by technological means without meaningful human involvement.

Bias: A systematic distortion that can produce unjustified or unequal outcomes.

Controller: The person or organisation determining the purposes and means of processing personal data.

Data minimisation: Limiting data to what is adequate, relevant and necessary.

Differential privacy: A formal privacy technique limiting the effect of one person’s data on an output.

Explainability: The ability to communicate how or why a system produces results.

Federated learning: Training across decentralised data locations without centralising all raw data.

Inference: Applying a trained model to produce an output from new input.

Meaningful human involvement: Human review that is informed, independent enough and able to alter an outcome.

Personal data: Information relating to an identified or identifiable natural person.

Profiling: Automated processing used to evaluate or predict personal characteristics or behaviour.

Pseudonymisation: Processing that separates direct identifiers while leaving a re-identification possibility.

Special-category data: Sensitive data receiving enhanced legal protection under data-protection law.

Synthetic data: Artificially generated data designed to resemble characteristics of real data.

Transparency: Providing understandable information about processing, purposes, effects and rights.

Training data: Data used to develop or adjust a model.

Inference data: Data supplied to a model when it is used in operation.

Review Questions

Short Answer Questions

1. Distinguish profiling from automated decision-making.
2. Why does removing a sensitive attribute not necessarily remove bias?
3. What is meant by meaningful human involvement?
4. Identify three data-protection principles relevant to AI.
5. Why can pseudonymised data remain personal data?

Long Answer Questions

1. Explain the privacy risks arising across the AI data lifecycle.
2. Discuss how transparency and explainability should be designed for an automated decision system.
3. Explain why historical recruitment data may produce discriminatory AI outcomes.
4. Describe the role of a data-protection impact assessment in an AI project.
5. Compare technical safeguards and organisational safeguards for AI governance.

Analytical / Application Questions

1. A bank uses transaction history and postcode to deny loans automatically. Analyse the privacy, profiling, proxy-discrimination and human-oversight issues.
2. An employer uses a language model to summarise employee complaints and train a future workplace tool. Identify lawful-purpose, confidentiality, retention and function-creep risks and propose controls.
3. A public authority introduces facial recognition for access to a service. Develop a proportionate assessment covering necessity, sensitive biometric data, accuracy, transparency, alternatives and challenge rights.

Multiple Choice Questions with Answers

Question 1

Which activity is most clearly profiling?

A. Encrypting a database

B. Predicting a person’s credit risk from personal data

C. Deleting duplicate files

D. Replacing a damaged server

Correct Answer: B. Predicting a person’s credit risk from personal data

Question 2

Which statement about pseudonymised data is correct?

A. They are always anonymous

B. They can never be used for profiling

C. They may remain personal data if re-identification is reasonably possible

D. They are exempt from all security duties

Correct Answer: C. They may remain personal data if re-identification is reasonably possible

Question 3

What best demonstrates meaningful human oversight?

A. A person automatically approves every model output

B. A person is named in the policy but never sees outputs

C. A trained reviewer evaluates the evidence and can change the result

D. A vendor promises that the model is accurate

Correct Answer: C. A trained reviewer evaluates the evidence and can change the result

Question 4

Why can removing a sensitive feature fail to eliminate discrimination?

A. Models cannot process numerical data

B. Other features may act as proxies for the sensitive characteristic

C. Sensitive features are required in every model

D. Removing features always increases accuracy

Correct Answer: B. Other features may act as proxies for the sensitive characteristic

Question 5

Which GDPR principle most directly requires limiting collection to necessary data?

A. Data minimisation

B. Storage limitation

C. Portability

D. Confidentiality

Correct Answer: A. Data minimisation

Question 6

What is a major risk of using historical hiring decisions as training labels?

A. Historical data are always anonymous

B. They may reproduce earlier institutional bias

C. They cannot contain personal data

D. They automatically satisfy fairness requirements

Correct Answer: B. They may reproduce earlier institutional bias

Question 7

What is the purpose of a data-protection impact assessment?

A. To guarantee that a model is profitable

B. To identify and mitigate risks to individuals before or during high-risk processing

C. To replace all cybersecurity testing

D. To eliminate the need for a lawful basis

Correct Answer: B. To identify and mitigate risks to individuals before or during high-risk processing

Question 8

In the differential-privacy expression Pr[M(D) ∈ S] ≤ e^ε × Pr[M(D′) ∈ S], what does D′ represent?

A. A neighbouring dataset differing in one person’s record

B. The final model version

C. A deleted database

D. A security password

Correct Answer: A. A neighbouring dataset differing in one person’s record

Question 9

Which statement best describes inference?

A. Applying a trained model to new input to produce an output

B. Deleting all training records

C. Writing a privacy notice

D. Assigning controller responsibility

Correct Answer: A. Applying a trained model to new input to produce an output

Question 10

Which is the strongest governance conclusion?

A. High overall accuracy proves legality

B. Publicly available data may always be reused

C. AI compliance requires legal, technical and organisational assessment

D. Human presence always defeats automated-decision rules

Correct Answer: C. AI compliance requires legal, technical and organisational assessment