14. AI-Enabled Software Engineering

Introduction and Context

Artificial intelligence is changing software engineering from a discipline concerned mainly with manually performed analysis, design, implementation, testing, deployment, and maintenance into a discipline in which human and machine capabilities are deliberately combined. AI-enabled software engineering does not mean that software development has become fully autonomous. Rather, it involves the use of machine learning, natural-language processing, generative models, intelligent agents, analytics, and automation to support engineering judgment and improve development outcomes.

The distinction between automation and AI assistance is important. Conventional automation executes predefined rules: a build server compiles code, a script runs tests, or a deployment pipeline provisions infrastructure. AI-enabled tools infer patterns from data, generate candidate artifacts, rank alternatives, detect anomalies, summarize information, and interact through natural language. These capabilities can reduce routine effort, but they can also introduce plausible errors, hidden bias, insecure code, intellectual-property concerns, and difficult questions of accountability.

For postgraduate software engineers, the central issue is therefore not whether AI should be used, but how it should be evaluated, governed, integrated, and measured. AI must be treated as an engineering component and socio-technical intervention, not as an unquestionable source of truth.

Learning Objectives

After studying this chapter, students should be able to:

1. Explain the role of AI across the software development life cycle.

2. Distinguish generative, predictive, analytical, and agentic AI applications in software engineering.

3. Evaluate AI-generated requirements, code, tests, and operational recommendations.

4. Design human-in-the-loop workflows with suitable verification and approval points.

5. Analyse risks involving security, privacy, bias, reliability, provenance, licensing, and accountability.

6. Select metrics for assessing productivity, quality, safety, and responsible adoption.

7. Recommend an organizational governance approach for AI-assisted engineering.

AI in the Software Development Life Cycle

AI can participate in nearly every life-cycle activity, although its role and acceptable level of autonomy differ by risk. A useful classification is as follows:

• Generative AI produces text, code, diagrams, test cases, documentation, queries, or configuration from prompts and contextual data.
• Predictive AI estimates defects, effort, failure probability, delivery risk, or likely user behaviour.
• Analytical AI classifies, clusters, summarizes, searches, and detects anomalies in engineering information.
• Agentic AI plans and performs a sequence of actions, such as modifying a repository, running tests, opening a change request, and responding to results.

These categories may overlap. An intelligent coding assistant may generate a function, use retrieval to find relevant project conventions, run tests, and revise its proposal. The more actions a system can perform without human intervention, the greater the need for authorization boundaries, monitoring, and rollback mechanisms.

Requirements Engineering

AI-assisted requirements engineering can transcribe meetings, identify actors and goals, extract candidate requirements from documents, detect duplicate statements, classify requirements, and generate questions about ambiguity. It can also convert natural-language requirements into user stories, acceptance criteria, use cases, or preliminary models.

The principal benefit is improved analysis of large and inconsistent information sources. The principal danger is mistaking linguistic fluency for requirements validity. A model may omit a stakeholder, infer an unstated business rule, or produce acceptance criteria that are testable but wrong. Requirements remain negotiated socio-technical agreements. Stakeholders must validate scope, priority, feasibility, assumptions, and non-functional properties such as safety, privacy, availability, and accessibility.

A sound workflow labels AI output as a candidate artifact. Each candidate should have a source reference, responsible reviewer, validation status, and decision history. Traceability should connect the requirement to stakeholders, design decisions, implementation elements, and tests.

Design and Architecture

AI tools can propose design alternatives, explain architectural patterns, generate interface specifications, identify dependencies, and review diagrams for inconsistency. They can assist with threat modelling by suggesting assets, trust boundaries, attack surfaces, and misuse cases.

Architectural recommendations must be assessed against system qualities and constraints. A generated recommendation may be elegant but incompatible with latency limits, regulatory obligations, operational skills, or an existing technology strategy. Architects should require explicit assumptions and compare alternatives using criteria such as modifiability, scalability, resilience, security, cost, and reversibility.

Coding and Code Review

Coding assistants generate boilerplate, complete functions, translate code between languages, explain unfamiliar modules, write queries, and suggest refactorings. They are particularly useful for repetitive transformations, documentation, test scaffolding, and exploration of unfamiliar APIs.

Generated code is not automatically correct, secure, efficient, or appropriately licensed. Common defects include incorrect edge-case handling, insecure deserialization, injection vulnerabilities, weak authentication logic, race conditions, improper error handling, and dependency misuse. Code review should therefore examine intent, not merely syntax. Compilation and passing tests provide evidence, but neither proves correctness.

A professional workflow combines prompt or context control, local build execution, static analysis, dependency scanning, unit and integration tests, peer review, and secure coding standards. High-risk code involving financial transactions, identity, safety, cryptography, or privacy requires stronger review and often prohibits unsupervised generation or direct production deployment.

Testing and Quality Assurance

AI can generate test cases from requirements or code, prioritize regression tests, identify likely fault locations, create test data, summarize failures, and detect visual or behavioural anomalies. Predictive models can help allocate testing effort toward components with a high historical defect probability.

Generated tests may reproduce the same misunderstanding as the generated implementation. This is an instance of correlated error: the implementation and its tests appear consistent because both were derived from an incorrect assumption. Independent oracles are therefore important. Test design should include domain experts, property-based testing, mutation testing, boundary analysis, security testing, exploratory testing, and production feedback where appropriate.

AI-generated test data also requires care. Real personal, financial, health, or confidential data should not be submitted to an external service without authorization. Synthetic data must be checked for realism, privacy leakage, and coverage of rare but important cases.

Operations and Reliability Engineering

In operations, AI supports log summarization, incident correlation, alert reduction, capacity forecasting, root-cause hypotheses, runbook retrieval, and anomaly detection. An operations assistant may answer questions about service health or recommend a rollback.

Operational recommendations should be grounded in current telemetry and clearly distinguish observed facts from inferred explanations. A recommendation to restart a service may hide a capacity, data-integrity, or security problem. For high-impact systems, AI should initially operate in advisory mode. Automated actions should be restricted by policy, tested in controlled environments, logged, reversible, and subject to rate limits and approval thresholds.

Maintenance and Evolution

Maintenance teams can use AI to recover undocumented architecture, summarize change impact, modernize legacy code, translate languages, identify dead code, and generate migration plans. These activities are valuable because maintenance often suffers from incomplete knowledge and scarce expertise.

However, modernization can unintentionally alter business rules. Before transformation, teams should establish behavioural baselines, characterization tests, dependency inventories, and rollback plans. AI can accelerate change, but responsibility for preserving externally visible behaviour remains with the maintenance organization.

A Human-Centred Engineering Model

An effective AI-enabled process assigns different responsibilities to the system and to people. AI is well suited to searching, drafting, pattern recognition, summarization, and proposing alternatives. Humans remain responsible for purpose, context, value judgments, accountability, exception handling, and acceptance of risk.

The human-in-the-loop model requires human approval before a consequential action. The human-on-the-loop model permits automated operation while humans monitor and intervene. The human-in-command model reserves the authority to stop, override, or constrain the system. These models should be selected according to impact, reversibility, uncertainty, and regulatory obligations.

A practical control framework contains five stages:

1. Define the task and risk level. Specify the intended use, affected parties, assets, and unacceptable outcomes.

2. Provide controlled context. Supply approved repositories, standards, schemas, and documentation while minimizing sensitive data.

3. Generate and record a proposal. Preserve prompts, model version, retrieved sources, output, and tool actions when appropriate.

4. Verify independently. Use tests, analysis, review, threat modelling, source comparison, and operational checks.

5. Approve, deploy, and monitor. Release through normal change controls and monitor both the software and the AI process.

This model transforms a conversational tool into a governed engineering activity.

Prompting, Retrieval, and Context Management

The quality of an AI response depends not only on the model but also on the task definition and context. Effective engineering prompts identify the role, objective, constraints, input format, desired output, evaluation criteria, and uncertainty requirements. Asking a tool to “write secure code” is weaker than specifying the threat model, language version, interfaces, failure behaviour, security standard, and tests required.

Retrieval-augmented generation supplies a model with relevant organizational documents or repository content at the time of a request. It can reduce unsupported invention and improve consistency with local standards. Nevertheless, retrieval systems can return obsolete, unauthorized, or misleading documents. Access control, document versioning, source ranking, and citation checking are essential.

Context management also involves data minimization. Prompts should exclude secrets, personal information, proprietary algorithms, and unnecessary source code. Enterprise deployments should define retention, training-use, encryption, access, and deletion policies for provider services.

Verification, Validation, and Provenance

Verification asks whether an artifact was built correctly according to specified conditions; validation asks whether the correct artifact was built for stakeholder needs. AI makes both questions more difficult because output can be persuasive without being dependable.

Verification practices include compilation, unit and integration testing, static analysis, formal checks where feasible, dependency and license scanning, performance measurement, and security review. Validation requires stakeholder review, realistic scenarios, usability evaluation, domain simulation, and comparison with business objectives.

Provenance is the record of where an artifact or recommendation came from. For AI-assisted engineering, provenance may include source documents, repository commits, model and tool versions, prompts, retrieved passages, generated output, reviewer decisions, and test evidence. Provenance supports debugging, audit, reproducibility, incident investigation, and intellectual-property analysis. It need not mean retaining sensitive prompts indefinitely; retention should follow policy and legal requirements.

Security, Privacy, Licensing, and Responsible Use

AI introduces risks in addition to ordinary software risks. Prompt injection can cause a tool to follow malicious instructions embedded in retrieved content. Data poisoning can influence a model or knowledge base. An agent with excessive repository or deployment permissions can amplify a small error into a major incident. Model outputs may reveal confidential information or reproduce protected material.

Security controls should include least privilege, sandboxing, network restrictions, secret scanning, isolated execution, approval gates, audit logs, dependency verification, and adversarial testing. Agent actions should be expressed through narrow, typed interfaces rather than unrestricted shell access wherever possible.

Privacy requires purpose limitation, consent or another lawful basis where applicable, minimization, retention controls, access management, and protection of personal data in prompts, training sets, logs, and generated outputs. Privacy impact assessment is appropriate when AI processes sensitive or large-scale personal information.

Licensing and provenance are also professional concerns. Generated code may resemble material from training or retrieval sources, and different tools impose different contractual conditions. Organizations should maintain an approved-tool list, review provider terms, scan dependencies, preserve attribution where required, and obtain legal guidance for uncertain cases. The fact that a model produced an output does not remove the organization's responsibility for its use.

Responsible adoption includes accessibility, non-discrimination, explainability proportionate to risk, contestability, and clear assignment of accountability. Affected users should not be denied meaningful recourse merely because an AI-assisted process was involved.

Measuring Value Without Encouraging Harm

AI adoption should be evaluated through a balanced set of measures rather than lines of code or raw generation volume. Useful dimensions include:

• Flow: lead time, review delay, deployment frequency, and time spent on repetitive work.
• Quality: escaped defects, change failure rate, reliability, test effectiveness, and maintainability.
• Safety and security: vulnerability density, policy violations, incidents, privacy events, and unauthorized actions.
• Human outcomes: developer comprehension, cognitive load, satisfaction, skill development, and inclusion.
• Economic value: effort avoided, infrastructure cost, support cost, and realized business benefit.
• Governance: percentage of outputs with provenance, review completion, approved-tool usage, and audit findings.

Metrics must be interpreted carefully. A higher deployment frequency is not beneficial if it increases failures. Faster code production may shift effort into review and maintenance. Teams should establish a baseline, run controlled pilots, examine qualitative evidence, and monitor long-term effects. Metrics should improve decisions rather than become targets that encourage unsafe behaviour.

Illustrative Case Study: AI-Assisted Claims Platform Modernization

Background

An insurance organization maintained a ten-year-old claims platform. Its maintenance team proposed using an AI coding assistant to translate selected modules, generate tests, and document undocumented business rules. The platform processed personal and financial information, so the organization classified it as high impact.

Situation and Problem

The existing code had limited test coverage and contained rules that were not fully represented in current requirements documents. A rapid, unsupervised translation could reduce short-term effort but might change eligibility calculations or expose confidential code to an external provider.

Analysis

The team performed a risk assessment and divided the system into low-, medium-, and high-risk components. It discovered that the high-risk components involved payment calculation and fraud-related decisions. The organization also identified data-protection, contractual, licensing, and operational-continuity requirements.

Approach and Solution

The team established the following controls:

• The assistant ran in an approved enterprise environment with restricted retention and no use of submitted code for general model training.
• Personal data and secrets were removed from prompts.
• The tool was initially limited to documentation, test scaffolding, and low-risk translation.
• Characterization tests were written around observed legacy behaviour before transformation.
• Every generated change required a named engineer, peer review, static analysis, dependency scanning, and regression testing.
• Payment and fraud modules required domain-expert approval and independent test design.
• Generated artifacts recorded source files, tool version, prompts, reviewer decisions, and test evidence.
• Deployment occurred through staged releases with monitoring and immediate rollback capability.

Outcome

The pilot reduced time spent on routine documentation and increased test coverage in the selected modules. It did not permit direct autonomous changes to production. Several generated translations failed edge-case tests, demonstrating why behavioural baselines and independent validation were necessary. The organization expanded the use case only after reviewing quality, security, privacy, and developer-comprehension measures.

Learning Points

The case shows that responsible AI adoption is a systems decision. Tool capability alone did not determine success. Scope selection, data controls, human expertise, provenance, independent tests, and reversible deployment were equally important. A slower initial pilot produced evidence that supported safer scaling.

Challenges and Common Mistakes

Several recurring mistakes weaken AI-enabled engineering:

• treating fluent output as verified output;
• sending confidential information to unapproved services;
• accepting generated tests that merely mirror implementation assumptions;
• granting agents broad write or deployment permissions;
• measuring productivity only by volume of generated code;
• failing to record model, prompt, source, and review information;
• assuming a general-purpose model understands local architecture or regulation;
• allowing organizational skills to decay through uncritical automation;
• overlooking accessibility, bias, and the interests of people affected by software decisions.

AI systems also exhibit non-determinism, distribution shift, hallucination, susceptibility to adversarial input, and uneven performance across domains and languages. Human review can itself become superficial when output volume is high. Governance must therefore address workload, expertise, escalation, and the design of review interfaces—not merely require a nominal approval.

Emerging Directions

Current development is moving toward repository-aware assistants, multimodal engineering tools, autonomous test and repair agents, AI-supported architecture analysis, and domain-specific models. These systems may increasingly plan multi-step work and interact with development platforms. This raises the importance of agent identity, delegated authority, action policies, execution sandboxes, and machine-readable audit trails.

Another important direction is evaluation engineering: organizations are constructing task-specific benchmarks, adversarial test sets, quality gates, and monitoring systems for AI components. Standardized assurance practices are also developing around risk management, secure development, privacy, and trustworthy AI. The most durable approach is likely to combine general governance principles with evidence from the particular system, domain, model, and use case.

Best-Practice Guidance

A mature organization should begin with valuable, bounded use cases rather than unrestricted deployment. It should maintain an AI use policy, approved tools, data classifications, training for engineers and reviewers, and a process for reporting incidents. Each use case should have an owner, risk classification, success measures, prohibited actions, review requirements, and retirement criteria.

AI output should enter the same configuration, review, testing, security, and change-management systems as human-produced output. High-impact decisions require stronger controls than low-risk drafting. Organizations should preserve meaningful human authority, periodically reassess models and providers, and evaluate whether the tool improves outcomes for users as well as developers.

Chapter Summary

AI-enabled software engineering applies generative, predictive, analytical, and agentic AI across requirements, architecture, coding, testing, operations, and maintenance. Its value lies in augmenting human capability, especially in search, drafting, pattern recognition, analysis, and repetitive work. Its risks arise from incorrect or biased output, correlated errors, security vulnerabilities, privacy leakage, licensing uncertainty, prompt injection, excessive agent authority, and weakened human judgment.

Reliable adoption requires a human-centred workflow: define purpose and risk, provide controlled context, record proposals and provenance, verify independently, approve through appropriate governance, and monitor outcomes. Requirements and architecture need stakeholder validation; code needs security and peer review; tests need independent oracles; operations need bounded and reversible actions; and maintenance needs behavioural baselines. Metrics should balance flow, quality, safety, human outcomes, economic value, and governance. Responsible AI is therefore not a separate activity but an extension of professional software engineering discipline.

Key Terms

AI-enabled software engineering: Use of AI technologies to support or perform software engineering activities under defined controls.

Generative AI: AI that creates new content such as text, code, tests, diagrams, or configuration.

Predictive AI: AI that estimates future events, risks, defects, or outcomes from data.

Agentic AI: AI capable of planning and executing multiple actions toward a goal.

Human-in-the-loop: A design in which human approval is required before a consequential action.

Human-on-the-loop: A design in which automated activity occurs while humans monitor and intervene.

Retrieval-augmented generation: Generation supported by retrieving relevant external or organizational information.

Prompt injection: Malicious or unintended instructions that manipulate an AI system through its input or retrieved content.

Provenance: Information documenting the origin, transformations, tools, and decisions associated with an artifact.

Correlated error: An error shared by an implementation and its validation artifacts because they rely on the same faulty assumption.

Characterization test: A test that records existing behaviour before a system is changed.

Data minimization: Limiting collected or supplied data to what is necessary for a defined purpose.

Model drift: Decline in model performance as data, context, or operating conditions change.

Least privilege: Granting only the permissions necessary for a task.

Guardrail: A technical or procedural constraint that limits unsafe AI behaviour.

AI hallucination: A plausible but unsupported or incorrect AI-generated statement or artifact.

Independent oracle: A validation source or method not derived from the same assumptions as the artifact being tested.

Change failure rate: The proportion of production changes that cause failure, rollback, or remediation.

Review Questions

Short Answer Questions

1. Distinguish generative AI from predictive AI in software engineering.

2. Why should AI-generated requirements be treated as candidate artifacts?

3. What is provenance, and why is it useful in AI-assisted development?

4. Explain the danger of correlated error in AI-generated testing.

5. State two controls appropriate for an AI agent with repository access.

Long Answer Questions

1. Explain how AI can support each major activity of the software development life cycle, and identify the principal risks at each stage.

2. Discuss the human-in-the-loop, human-on-the-loop, and human-in-command models. When is each appropriate?

3. Develop a governance framework for using generative AI to produce production code in a regulated organization.

4. Explain how security, privacy, licensing, and provenance concerns interact in enterprise AI adoption.

5. Discuss why AI adoption should be evaluated using balanced engineering, human, economic, and governance metrics.

Analytical / Application Questions

1. A development team wants an autonomous agent to modify code, run tests, and merge pull requests. Design risk controls, approval points, and metrics for a six-month pilot.

2. An AI assistant recommends restarting a failing payment service after detecting anomalous logs. Analyse what evidence should be checked before action and propose a safe operational workflow.

3. A company submits customer support transcripts and legacy source code to a public AI service. Evaluate the risks and recommend a compliant alternative architecture and process.

22. Multiple Choice Questions with Answers

Question 1

Which statement best describes AI-enabled software engineering?

A. Replacing all software engineers with autonomous systems

B. Using AI capabilities to support engineering activities under appropriate human and technical controls

C. Using only rule-based build and deployment scripts

D. Generating code without testing because models learn from existing software

Correct Answer: B. Using AI capabilities to support engineering activities under appropriate human and technical controls

Question 2

Why should AI-generated requirements be treated as candidate artifacts?

A. AI cannot process natural language

B. Requirements never need stakeholder approval

C. Fluent output may omit stakeholders, invent assumptions, or misrepresent business rules

D. Candidate artifacts cannot be stored in repositories

Correct Answer: C. Fluent output may omit stakeholders, invent assumptions, or misrepresent business rules

Question 3

Which practice most directly addresses correlated error in AI-generated testing?

A. Generating more tests from the same implementation prompt

B. Using independent domain review and test oracles

C. Removing all regression tests

D. Accepting tests whenever they pass

Correct Answer: B. Using independent domain review and test oracles

Question 4

An AI coding assistant proposes insecure deserialization code that compiles successfully. What is the most appropriate response?

A. Accept it because compilation proves correctness

B. Deploy it and wait for production feedback

C. Review it using security analysis, tests, and secure coding standards

D. Delete all AI-generated code from the project

Correct Answer: C. Review it using security analysis, tests, and secure coding standards

Question 5

What is the main purpose of retrieval-augmented generation in an engineering context?

A. To eliminate the need for access control

B. To provide relevant organizational information as context for generation

C. To guarantee that every generated answer is correct

D. To permit unrestricted access to all repositories

Correct Answer: B. To provide relevant organizational information as context for generation

Question 6

Which control best limits the impact of an AI operations agent making an incorrect recommendation?

A. Granting unrestricted production permissions

B. Removing audit logs to improve speed

C. Using bounded permissions, approval gates, monitoring, and rollback

D. Allowing the agent to modify its own policies

Correct Answer: C. Using bounded permissions, approval gates, monitoring, and rollback

Question 7

Which item is an example of provenance?

A. The colour of an application's user interface

B. The model version, source documents, prompt, reviewer decision, and test evidence for a generated change

C. The number of employees in an organization

D. The average screen size of users

Correct Answer: B. The model version, source documents, prompt, reviewer decision, and test evidence for a generated change

Question 8

Why is measuring only the volume of generated code a poor adoption metric?

A. Generated code cannot be counted

B. More code always reduces productivity

C. It may reward output while ignoring defects, security, maintenance effort, and user outcomes

D. Code volume is unrelated to software engineering

Correct Answer: C. It may reward output while ignoring defects, security, maintenance effort, and user outcomes

Question 9

In the illustrative claims-platform case, why were characterization tests created before transformation?

A. To document and preserve observed legacy behaviour

B. To replace all domain experts

C. To avoid using version control

D. To increase the amount of generated code

Correct Answer: A. To document and preserve observed legacy behaviour

Question 10

Which principle is most appropriate when sending information to an external AI service?

A. Submit all available data to improve response quality

B. Use data minimization and exclude sensitive information unless authorized and protected

C. Disable access control because prompts are temporary

D. Assume provider terms are identical across services

Correct Answer: B. Use data minimization and exclude sensitive information unless authorized and protected