AI-Enabled Quality and Emerging Risks

Introduction

Artificial intelligence (AI) is increasingly embedded in software quality management. Modern tools can generate test cases, identify suspicious code changes, summarise defects, recommend fixes, detect anomalous project trends and estimate the likelihood of failure. These capabilities can improve speed and coverage, but they do not remove the need for professional judgement. AI systems learn from data, operate with uncertainty and may produce plausible but incorrect outputs.

For software project managers, the central issue is therefore not whether AI should be used, but how it can be used safely, transparently and proportionately. Quality remains a managed property of the product and its development process. AI is an assisting capability within that process, not an independent source of assurance.

This chapter examines practical uses of AI in quality and risk management and the controls needed to address bias, hallucination, data leakage, intellectual-property exposure, limited explainability and weak model governance.

Learning objectives

By the end of this chapter, the reader should be able to:

1. Explain how AI supports software testing, defect prediction, code review and project-risk sensing.
2. Distinguish automation, machine learning, generative AI and predictive analytics in quality work.
3. Assess the reliability and limitations of AI-generated recommendations.
4. Identify risks involving bias, hallucination, confidentiality, intellectual property and explainability.
5. Design proportionate controls for human oversight, validation, monitoring and model governance.
6. Decide when AI output is suitable for assistance, recommendation or controlled automation.

1. AI in the quality-management context

AI refers broadly to computational systems that perform tasks associated with perception, prediction, language, reasoning or decision support. The term covers several different technologies:

• Rule-based automation executes explicitly programmed rules, such as checking whether a required field is empty.
• Machine learning identifies patterns in data and uses them to classify, rank or predict new cases.
• Generative AI produces text, code, test data, images or other content in response to prompts and context.
• Large language models generate language and code by modelling statistical relationships in large datasets.
• Anomaly detection identifies observations that differ substantially from an established pattern.

These categories should not be treated as interchangeable. A deterministic static-analysis rule may produce repeatable results, whereas a generative model may produce different answers to similar prompts. A defect-prediction model may show statistical association without proving that a particular file contains a defect.

A useful quality principle is:

AI output is evidence to evaluate, not a verdict to accept automatically.

The quality manager should define the intended decision, the consequences of error, the evidence required and the person accountable for the final decision.

2. Practical uses across the software life cycle

2.1 AI-assisted test generation

AI can generate test ideas from requirements, user stories, API specifications, source code or historical defects. It may propose:

• positive and negative functional tests;
• boundary-value and equivalence-partition tests;
• API request and response combinations;
• regression tests based on changed components;
• synthetic test data;
• property-based test hypotheses;
• accessibility, security and resilience scenarios.

For example, from a requirement stating that a customer may transfer between £1 and £5,000 per day, an AI tool might suggest values below, at and above each boundary, together with invalid currencies, repeated submissions and interrupted network connections.

AI-generated tests are useful for breadth and ideation, but they can reproduce ambiguity in the requirement. They may also generate tests that merely repeat implementation details, omit business-critical risks or contain assertions too weak to detect failure. A human tester must check the oracle: the rule used to determine whether the result is correct.

A practical workflow is:

1. provide the tool with an approved requirement or interface contract;
2. request test conditions rather than immediately requesting executable code;
3. review coverage, assumptions, data sensitivity and expected outcomes;
4. implement selected tests in the controlled test repository;
5. execute them in a safe environment;
6. review failures independently; and
7. record the origin and approval status of generated tests.

2.2 Defect prediction and prioritisation

Defect-prediction models use historical information to estimate which files, services or changes deserve additional attention. Candidate features include code churn, complexity, dependency centrality, past defects, review history, test failures and frequency of change.

A model may support resource allocation by ranking components for review. It should not be interpreted as proving that a highly ranked component is defective or that a low-ranked component is safe. Predictions are affected by the quality and representativeness of historical data. If a team tested one subsystem more rigorously than another, the data may make the first subsystem appear more defective even when its underlying quality is better.

Common evaluation measures include:

• Precision: the proportion of items flagged by the model that are actually defective.
• Recall: the proportion of all defective items that the model successfully flags.
• False-positive rate: the proportion of non-defective items incorrectly flagged.
• Calibration: the degree to which predicted probabilities correspond to observed frequencies.

For a binary classification task, if TP denotes true positives, FP false positives and FN false negatives, then:

These measures assume clearly labelled outcomes and a defined decision threshold. They should be interpreted with the operational cost of errors. In safety-critical software, missing a defect may be much more serious than reviewing an additional file. In a rapidly changing project, excessive false positives may cause alert fatigue and cause engineers to disregard the tool.

2.3 AI-assisted code review

AI code-review tools can identify likely defects, insecure patterns, duplicated logic, missing tests, readability problems and possible violations of coding standards. Generative tools can also explain code or propose a patch.

The principal risk is over-trust. A fluent explanation may conceal an incorrect diagnosis, and a suggested patch may introduce a vulnerability or alter business behaviour. AI review should complement, not replace, compilation, static analysis, automated tests, peer review and security review.

A controlled code-review process should require that:

• the tool receives only the minimum necessary code and metadata;
• generated findings are linked to a specific file and line or rule;
• the reviewer verifies the finding against requirements and tests;
• proposed patches are independently reviewed;
• secrets and personal data are excluded from prompts;
• the tool’s version and configuration are recorded for material decisions.

2.4 Risk sensing and project intelligence

AI can analyse issue trackers, build results, delivery trends, dependency alerts, review comments and communication patterns to identify emerging risks. Examples include increasing rework, repeated build failures, unresolved high-severity defects, unusual deployment frequency or concentration of knowledge in one team member.

Risk sensing should be treated as an early-warning system. A signal is not itself a risk statement. The project manager must investigate the cause, estimate probability and impact, assign ownership and select a response. AI should not silently infer individual performance from private communications or use opaque scores for employment decisions.

A useful risk record contains:

• the event that may occur;
• its cause and potential consequence;
• evidence supporting the assessment;
• probability and impact assumptions;
• owner, response and due date;
• current status and review history.

3. A decision model for adopting AI

Before adoption, the team should classify the proposed use by consequence and autonomy.

The team should ask five questions:

1. What decision will the output influence?
2. What is the cost of a false positive and a false negative?
3. What data does the system require, and may that data legally and ethically be used?
4. Can a qualified person understand and challenge the output?
5. What happens if the model is unavailable, wrong or compromised?

A simple risk-prioritisation score can be used for operational triage:

Here, likelihood and impact must use defined ordinal scales, such as 1 to 5. The product is a prioritisation aid, not a probability calculation. Multiplication assumes that the organisation has deliberately chosen scales whose relative ordering is meaningful; it does not establish a universal measure of risk.

4. Emerging AI-specific risks

4.1 Bias and unfair performance

Bias occurs when an AI system produces systematically poorer or less favourable results for particular groups, contexts or classes of software. Sources include unrepresentative training data, historical process bias, unequal labelling quality, proxy variables and feedback loops.

In quality management, bias may appear when a defect model under-flags modules written in a newer language because historical examples are dominated by older systems. A test-generation model may produce weak accessibility tests because accessibility examples were scarce in its source material. A risk-sensing system may interpret a team’s communication style as a sign of low productivity.

Controls include representative evaluation datasets, subgroup performance analysis, review of proxy variables, transparent thresholds and periodic revalidation. Fairness is context-dependent: the relevant groups and acceptable error differences must be defined with domain experts and affected stakeholders.

4.2 Hallucination and unreliable generation

A hallucination is a confident-looking output that is unsupported, fabricated or incorrect. Examples include invented API functions, false references, nonexistent vulnerabilities or test assertions based on an incorrect interpretation of a requirement.

Controls include retrieval from approved sources, constrained prompts, schema validation, compilation, execution of generated tests, citation checking and human approval. The more consequential the use, the stronger the verification must be. Fluency, confidence or a high numerical score from the model is not evidence of truth.

4.3 Data leakage and confidentiality

Prompts and uploaded files may be retained, logged, used for service improvement or exposed through a misconfigured integration. Sensitive material may include source code, credentials, customer records, security findings, unreleased product plans and personal data.

A project should establish a data-classification policy for AI use. Public data, internal data, confidential source code and regulated data require different controls. Important measures include enterprise-approved tools, contractual restrictions on provider training, encryption, access control, redaction, retention limits, tenant separation and audit logging. Secrets must never be placed in prompts.

4.4 Intellectual property and licensing

Generated code may resemble training material or may be subject to licence obligations that the user cannot easily identify. The organisation must distinguish ownership, permitted use and attribution requirements. Existing open-source policy, software composition analysis and legal review remain necessary. AI assistance does not transfer responsibility for compliance to the tool provider.

4.5 Explainability and contestability

A score without an understandable rationale is difficult to use responsibly. Explainability can involve feature contributions, comparable examples, rule traces, confidence intervals or a plain-language statement of limitations. No single explanation method is universally adequate.

Contestability is equally important: an engineer should be able to challenge a finding, provide contrary evidence and request a human decision. Explanations should be available to the people making the decision, not only to the model developer.

4.6 Model drift and dependency risk

Software projects, coding practices, threat patterns and team structures change. A model trained on yesterday’s data may become less accurate. Drift can concern the input distribution, the relationship between inputs and outcomes, or the meaning of the target label.

The organisation should monitor performance over time, compare predictions with verified outcomes, review threshold behaviour and define retraining or retirement criteria. It should also plan for provider outages, pricing changes, model replacement and loss of access to a proprietary service.

5. Model governance across the life cycle

Model governance is the set of responsibilities, controls and records that make AI use accountable. It should cover both internally developed models and externally supplied tools.

Before deployment

The team should document the intended purpose, prohibited uses, owner, data sources, target users, performance measures, security requirements and residual risks. A privacy and security assessment may be required. Procurement should examine provider terms, data handling, audit rights, service levels, incident notification and model-change practices.

During validation

Validation data should be separated from development data where practical. Tests should include normal cases, edge cases, adversarial inputs and relevant subgroups. The team should evaluate not only average accuracy but also the consequences of different error types. For generative systems, validation should include factuality, refusal behaviour, prompt-injection resistance and harmful-output testing.

During operation

Operational controls include role-based access, logging, version identification, approval gates, output sampling, incident reporting, drift monitoring and periodic review. A material model change should trigger impact assessment rather than being treated as an ordinary software update.

At retirement

Retirement requires removal or restriction of access, preservation of required records, secure deletion of sensitive data and migration to a replacement process. The organisation should document why the model was retired and whether its outputs remain in products, tickets or decisions.

6. Human oversight and assurance

Human-in-the-loop is not a sufficient control merely because a person clicks an approval button. Effective oversight requires competence, time, authority and access to relevant evidence. Reviewers must be able to reject an output without unreasonable penalty.

Three practical safeguards are especially important:

1. Independent verification: use tests, static analysis, peer review or domain evidence that does not simply repeat the model’s reasoning.
2. Traceability: record input source, model or tool version, prompt or configuration where material, output, reviewer and decision.
3. Fail-safe operation: when confidence is low, data is incomplete or the system is unavailable, revert to a defined manual or deterministic process.

Assurance should be risk-based. A tool that drafts low-impact test descriptions needs less governance than a tool that recommends release approval for medical software.

7. Illustrative case: AI-assisted regression planning

Background

A fictional software company, Northbridge Payments, maintains a payment platform with several thousand automated regression tests. Releases are frequent, but the full suite takes eight hours. The company considers an AI system that ranks tests according to the files changed, historical failures, dependency relationships and recent production incidents.

Problem

The delivery team wants faster feedback. The quality manager is concerned that the ranking model may omit rare but severe payment failures. Historical data also contains more tests from the card-payment service than from the refund service, which was tested less consistently.

Analysis

The proposed system is advisory, not an autonomous release gate. The team defines success as reducing execution time while maintaining coverage of high-severity business risks. It separates the evaluation period from the data used to tune the ranking model. It measures precision of selected tests, recall of historically detected defects and coverage of mandatory risk-based tests.

The team discovers that the model gives low priority to refund tests because few historical failures exist. Investigation shows that this reflects limited testing, not low risk. The team therefore adds a mandatory business-risk rule requiring refund, chargeback, authentication and reconciliation tests in every release, regardless of model ranking.

Approach and controls

The organisation:

• removes customer data and secrets from the model inputs;
• uses the model only to rank optional tests;
• retains a deterministic safety set;
• reviews missed defects after every release;
• monitors performance separately for each payment service;
• records model version and ranking rationale;
• provides a manual override for release managers; and
• reassesses the model after major architecture or process changes.

Outcome

The selected suite provides faster feedback, while mandatory tests preserve coverage of critical functions. The model is not judged solely by average time saved. Its continued use depends on evidence that it does not increase the risk of undetected severe defects.

Learning points

The case illustrates four principles: historical data may encode process weaknesses; risk-based controls must supplement statistical ranking; advisory AI is safer than unreviewed automation; and performance must be evaluated by consequence, not convenience alone.

8. Implementation roadmap for a project team

A certificate-level project manager can introduce AI responsibly through a staged roadmap:

1. Identify a bounded use case. Begin with low- or moderate-consequence assistance, such as test-idea generation or log summarisation.
2. Define acceptance criteria. Specify accuracy, coverage, response time, data restrictions and human-approval requirements.
3. Classify data. Decide what may be shared, what must be redacted and what is prohibited.
4. Pilot in a controlled environment. Compare AI-supported work with the existing process using representative examples.
5. Record failures. Hallucinations, omissions, leakage incidents and misleading explanations are valuable governance evidence.
6. Train users. Teach prompting, verification, secure handling and escalation.
7. Monitor and review. Establish owners, dashboards, review intervals and retirement triggers.
8. Scale selectively. Expand only when evidence supports the new level of autonomy.

9. Best practices and limitations

Best practice is to use AI where it increases human capability without obscuring accountability. Teams should prefer narrow, testable tasks over vague claims that AI will “improve quality.” They should preserve deterministic checks for rules that must be applied consistently, maintain independent test oracles, use least-privilege access and involve security, legal, privacy and domain specialists when appropriate.

AI cannot compensate for unclear requirements, weak test design, missing ownership or poor engineering discipline. It can amplify existing data and process problems. Cost, vendor lock-in, environmental impact, changing model behaviour and staff over-reliance are additional concerns. Generative AI may also create a false impression of productivity if review and correction effort is ignored.

The most defensible position is neither unrestricted adoption nor blanket rejection. It is controlled experimentation followed by evidence-based deployment, continuous monitoring and the ability to withdraw the system when its risks exceed its value.

Chapter Summary

AI can assist software quality through test generation, defect prediction, code review and project-risk sensing. These uses differ in technology, evidence requirements and consequences of error. AI-generated content and predictions must be validated because models can hallucinate, inherit bias, leak confidential information, create intellectual-property concerns and become inaccurate as projects change.

Responsible use requires a defined purpose, classified data, measurable acceptance criteria, human accountability, independent verification, traceability, security controls and monitoring. Model governance should cover selection, validation, operation, change and retirement. Risk-based deployment is essential: low-impact assistance may require ordinary review, while high-impact automation requires stronger assurance and explicit authorisation. AI should improve quality-management decisions without replacing professional judgement.

Key Terms

Artificial intelligence: Computational capability that performs tasks associated with perception, prediction, language or reasoning.

Machine learning: Methods that learn patterns from data to support prediction or classification.

Generative AI: AI that creates new text, code, test data or other content.

Large language model: A model trained to generate and analyse language and often computer code.

Test oracle: A rule, specification or reference used to determine whether a test result is correct.

Defect prediction: Estimation of where defects are more likely to occur.

Precision: The proportion of flagged cases that are true positives.

Recall: The proportion of true positive cases that are successfully flagged.

Calibration: Agreement between predicted probabilities and observed frequencies.

Hallucination: A plausible-looking but unsupported or incorrect AI output.

Bias: Systematic distortion that produces unfair or inaccurate results for some groups or contexts.

Data leakage: Unauthorised exposure or use of sensitive information.

Intellectual property: Legally protected creations, including source code, designs and documentation.

Explainability: The extent to which an AI result and its limitations can be understood.

Contestability: The ability to challenge, correct or appeal an AI-supported decision.

Model drift: Decline in model relevance or performance as data or conditions change.

Human oversight: Competent human review, intervention and accountability for AI-supported decisions.

Model governance: Policies, responsibilities and controls governing an AI system throughout its life cycle.

Risk-based testing: Allocating testing effort according to likelihood and consequence of failure.

Fail-safe operation: Reverting to a controlled alternative when an AI system is unavailable or unreliable.

Review Questions

Short Answer Questions

1. What is the difference between AI-assisted test generation and autonomous testing?
2. Define precision and recall in defect prediction.
3. Why can historical defect data produce biased predictions?
4. Identify two controls for preventing confidential data leakage.
5. What is meant by model drift?

Long Answer Questions

1. Explain how AI can support software quality across testing, code review and risk sensing. Discuss the limitations of each use.
2. Describe a model-governance life cycle from selection to retirement.
3. Explain why human oversight requires more than a final approval click.
4. Discuss hallucination, intellectual-property risk and explainability in AI-assisted code review.
5. Compare assistive, advisory and controlled-automation uses of AI in a software project.

Analytical / Application Questions

1. A defect-prediction tool flags 40 modules, of which 10 contain confirmed defects. It missed 5 defects found elsewhere. Calculate its precision and recall, and explain whether those figures are sufficient to approve automatic release blocking.
2. An AI test-ranking tool consistently deprioritises a recently introduced subsystem because it has no historical defects. Analyse at least three possible causes and propose controls.
3. A developer wants to upload a complete proprietary repository, including customer records and credentials, to a public coding assistant. Prepare a risk-based response covering data protection, security, intellectual property, verification and an acceptable alternative workflow.

Multiple Choice Questions with Answers

Question 1

Which statement best describes the role of AI-generated test cases?

A. They replace the need for test oracles.

B. They provide candidate tests that require review and validation.

C. They guarantee complete requirements coverage.

D. They are always more reliable than manually designed tests.

Correct Answer: B. They provide candidate tests that require review and validation.

Question 2

If a defect model flags 20 modules and 8 are confirmed defective, what is its precision?

A. 20%

B. 40%

C. 60%

D. 80%

Correct Answer: B. 40%

Question 3

Which situation is an example of AI hallucination?

A. A tool identifies a known coding standard violation.

B. A tool ranks files using recorded historical data.

C. A tool invents an API function that does not exist.

D. A tool records the model version in an audit log.

Correct Answer: C. A tool invents an API function that does not exist.

Question 4

Which control most directly reduces the risk of uploading confidential source code to an AI service?

A. Increasing the model temperature

B. Using approved tools with access and retention controls

C. Allowing all developers to share the same account

D. Disabling software testing

Correct Answer: B. Using approved tools with access and retention controls

Question 5

Why should a defect-prediction score not be treated as proof of a defect?

A. Predictions show statistical likelihood rather than confirmed evidence.

B. Prediction models cannot process numerical data.

C. Historical data is never useful.

D. All model outputs are deterministic rules.

Correct Answer: A. Predictions show statistical likelihood rather than confirmed evidence.

Question 6

What is model drift?

A. A deliberate change to a project schedule

B. A decline in model relevance as conditions or data change

C. A method for encrypting source code

D. A technique for generating boundary tests

Correct Answer: B. A decline in model relevance as conditions or data change

Question 7

Which practice best supports explainability and contestability?

A. Hiding the model version from reviewers

B. Preventing engineers from challenging predictions

C. Recording evidence and allowing a qualified person to review and reject findings

D. Using only the most complex model available

Correct Answer: C. Recording evidence and allowing a qualified person to review and reject findings

Question 8

In the illustrative Northbridge Payments case, why were mandatory tests retained?

A. The model could not rank any tests.

B. Critical business risks required coverage regardless of statistical ranking.

C. The company wanted to increase execution time.

D. Mandatory tests eliminate all model bias.

Correct Answer: B. Critical business risks required coverage regardless of statistical ranking.

Question 9

Which use normally requires the strongest assurance?

A. Drafting an internal meeting summary

B. Suggesting test names

C. Recommending whether safety-critical software may be released

D. Formatting a defect report

Correct Answer: C. Recommending whether safety-critical software may be released

Question 10

What is the main purpose of an independent test oracle?

A. To make prompts longer

B. To determine correctness without relying solely on the generated output

C. To reduce source-code ownership

D. To guarantee zero false positives

Correct Answer: B. To determine correctness without relying solely on the generated output