Analytics, AI and Intelligent System Design

Introduction

Information systems increasingly do more than record transactions. They analyse patterns, recommend actions, predict events and, in some settings, execute decisions automatically. A university admissions system may identify incomplete applications; a bank may estimate credit risk; a hospital may prioritize cases; and an online retailer may recommend products. These applications are often described as analytics, artificial intelligence (AI), or intelligent systems. However, their success depends less on selecting an impressive algorithm than on analysing the complete socio-technical system: its purpose, users, data, decision rules, risks, controls and operating environment.

System analysis and design therefore extends beyond functional requirements such as “calculate a score.” Analysts must determine what decision is being supported, whose interests are affected, how data are acquired and transformed, how model outputs will be interpreted, what happens when the model is uncertain, and who remains accountable. An intelligent system is not merely a model embedded in software. It is an organized combination of people, processes, data, technology, governance and feedback.

This chapter presents a requirements-centred approach to designing analytics and AI systems. It covers data pipelines, descriptive and predictive analytics, decision support, the model lifecycle, explainability, bias, human oversight, automation, monitoring and responsible AI governance.

Learning Objectives

After studying this chapter, a student should be able to:

1. Distinguish analytics, machine learning, AI and intelligent decision-support systems.
2. Analyse business and social requirements for an analytics or AI initiative.
3. Describe a reliable data pipeline and assess data quality and provenance.
4. Explain the stages of the model lifecycle from problem definition to retirement.
5. Select suitable performance, fairness and operational measures.
6. Evaluate explainability, bias, human oversight and automation risks.
7. Design monitoring and governance controls for responsible operation.
8. Apply these ideas to a realistic system-analysis case.

1. From Information Systems to Intelligent Systems

Traditional information systems generally follow explicit rules. A payroll system applies stated formulas to employee records, while an inventory system updates stock after a sale. Analytics systems use data to describe, diagnose, predict or prescribe.

Artificial intelligence is a broad term for systems performing tasks associated with human cognitive capabilities, such as perception, language processing, reasoning or learning. Machine learning is a family of methods in which a system derives patterns from data rather than relying entirely on manually written rules. The terms overlap but are not identical: a dashboard is analytics but not necessarily AI; a machine-learning classifier is AI-related but does not independently constitute a complete intelligent system.

An intelligent system has at least five connected layers:

1. Purpose and decision context: the organizational or social problem.
2. Data and knowledge: source data, definitions, labels, domain rules and context.
3. Inference or analytical logic: statistical models, machine-learning models, rules or optimization.
4. Interaction and action: interfaces, recommendations, alerts and downstream workflows.
5. Governance and accountability: permissions, auditability, human review, monitoring and remedies.

Failure at any layer can make the whole system unsafe or ineffective. A highly accurate model trained on the wrong target can support a harmful decision. A sound model can also fail if users misunderstand its output or if data are unavailable at the time of decision.

2. Requirements Analysis for Analytics and AI

2.1 Begin with the decision, not the algorithm

A weak requirement says, “Build an AI model to identify high-risk customers.” A stronger requirement identifies the decision and its consequences: “At the time an application is reviewed, provide a risk estimate to a trained officer, who may request additional evidence but must record the final decision and reason.” The second statement clarifies timing, user, action, accountability and the model’s role.

A requirements analyst should ask:

• What decision or process is being improved?
• Who is affected directly and indirectly?
• What is the cost of false positives and false negatives?
• What information is available before the decision, rather than after it?
• Is the output a description, prediction, recommendation or automatic action?
• What alternatives exist if the model is unavailable or uncertain?
• What legal, contractual, ethical or sector-specific constraints apply?

2.2 Stakeholder and impact analysis

Stakeholders include data subjects, operational users, managers, data owners, developers, auditors, regulators and people who may be denied a service because of the system. Their interests may conflict. A manager may prefer high automation, while a caseworker may need explanations and override authority. A data subject may require access, correction or appeal.

An impact assessment should consider severity, likelihood, reversibility and scale. Decisions affecting employment, education, health, housing, insurance or essential services deserve stronger controls than low-risk personalization. Analysts should document intended use, prohibited use, affected groups, assumptions and residual risks before implementation.

2.3 Functional and non-functional requirements

Functional requirements specify what the system does: ingest records, calculate a score, display evidence, route cases, record overrides or generate reports. Non-functional requirements specify qualities and constraints:

• Accuracy and usefulness: performance appropriate to the decision.
• Timeliness: maximum acceptable response time or data freshness.
• Availability and resilience: behaviour during outages and degraded operation.
• Security and privacy: authentication, authorization, encryption, minimization and retention.
• Explainability: information needed by users and affected persons.
• Fairness: defined evaluation across relevant groups.
• Auditability: reproducible records of inputs, model version and action.
• Accessibility: usable interaction for people with different abilities.
• Maintainability: ability to update data, rules and models safely.

Requirements should be testable. “The model must be fair” is incomplete. A better requirement might specify the protected groups, evaluation population, fairness measures, acceptable thresholds, review frequency and escalation process. No single fairness measure is universally correct; the appropriate choice depends on the decision, evidence and governing policy.

3. Data as a System Resource

3.1 The analytical data pipeline

A typical pipeline contains the following stages:

Figure: A data-to-decision pipeline with feedback and monitoring.

1. Source identification: operational databases, sensors, forms, documents, APIs or external datasets.
2. Ingestion: movement into analytical storage, either in batches or continuously.
3. Storage and preparation: validation, cleaning, joining, transformation and feature construction.
4. Training or inference: model development and production prediction.
5. Decision integration: presentation to a user, workflow or automated service.
6. Outcome capture: recording the action and later result for evaluation and learning.

Each transition requires ownership and controls. Data lineage should answer where a field originated, how it changed, when it was collected and which model used it.

3.2 Data quality and leakage

Important quality dimensions include accuracy, completeness, consistency, timeliness, uniqueness, validity and representativeness. Quality is purpose-dependent. A missing postal code may be tolerable for a clinical alert but unacceptable for geographical service planning.

Data leakage occurs when training information contains knowledge that would not be available at the real decision time, or when the target outcome is indirectly included in a predictor. Leakage can produce excellent test results but poor operational performance. Analysts must define a time boundary and ensure that every feature respects it.

A simple classification evaluation uses a confusion matrix. If a binary system predicts a positive condition, true positives (TP) are correctly identified positives, true negatives (TN) are correctly identified negatives, false positives (FP) are negative cases incorrectly classified as positive, and false negatives (FN) are positive cases missed by the system.

Accuracy is the proportion of all predictions that are correct. Precision is the proportion of predicted positives that are actually positive. Recall, also called sensitivity, is the proportion of actual positives detected. These measures assume a defined binary classification and a specified decision threshold. They may be misleading with highly imbalanced classes; analysts should also consider the costs of errors and appropriate measures such as specificity, calibration or area under a curve.

Worked example. Suppose TP = 80, TN = 850, FP = 50 and FN = 20. There are 1,000 cases. Accuracy is (80 + 850) ÷ 1,000 = 0.93, or 93%. Precision is 80 ÷ (80 + 50) = 80 ÷ 130 ≈ 61.5%. Recall is 80 ÷ (80 + 20) = 80%. The apparently high accuracy does not mean that every positive alert is reliable; precision is much lower because false positives are substantial.

4. The Model Lifecycle

A responsible model lifecycle is iterative rather than a one-time development project.

Problem framing

Define the target population, decision, prediction horizon, intervention and success criteria. Ask whether prediction is necessary. Sometimes a clear policy rule, additional staffing or better data collection solves the problem more transparently.

Data and label design

Identify data sources and the meaning of the outcome label. Historical labels may reflect previous human bias, unequal access or inconsistent measurement. A model trained to predict “past approval” may reproduce past decisions rather than estimate genuine need or merit.

Development and validation

Separate development data from evaluation data. Where data are time-dependent, a time-based split may be more realistic than a random split. Compare with a simple baseline, document features and preserve experiment records. Validation should test not only average performance but also subgroups, unusual cases, missing data and distribution changes.

Deployment

Deployment includes model packaging, access controls, versioning, rollback, user training, documentation and integration with workflow. A model should not be released merely because it performs well in a development environment.

Operation and retirement

Monitor predictions, input quality, latency, user overrides, outcomes, subgroup performance and incidents. Retraining must be controlled; new data do not automatically justify a new model. Retirement is appropriate when the purpose changes, performance degrades, risks become unacceptable or a simpler solution becomes available.

Model documentation should include purpose, intended users, training data, exclusions, limitations, evaluation results, fairness analysis, thresholds, dependencies, version history and contact owners. For AI systems using generative models, documentation should additionally cover prompts, retrieval sources, output constraints, evaluation for hallucination and procedures for handling sensitive information.

5. Explainability and Interpretability

Interpretability concerns how readily a model’s operation can be understood. Explainability concerns communicating why a particular output was produced. A linear model may be comparatively interpretable, whereas a complex neural network may require post-hoc explanation techniques. Neither transparency nor an explanation automatically proves that a decision is valid.

Different stakeholders need different explanations:

• A developer needs feature behaviour, errors and technical diagnostics.
• An operational user needs concise, actionable reasons and uncertainty.
• An affected person may need the principal factors, relevant evidence and a route to correction or appeal.
• An auditor needs reproducible records, model version and governance evidence.

Explanations should be faithful, understandable and appropriate to the decision. Feature importance can describe association without proving causation. A plausible-sounding explanation can be misleading if it is not faithful to the model. Analysts should test explanations with representative cases and avoid revealing security-sensitive details that would enable manipulation.

6. Bias, Fairness and Inclusion

Bias may enter through problem framing, sampling, measurement, labels, features, modelling, thresholds, deployment or feedback. Excluding a protected attribute from the model does not eliminate bias because other variables may act as proxies. Conversely, collecting protected-attribute information under strict controls may be necessary to evaluate disparate outcomes.

Fairness is not a single technical property. Common questions include:

• Are error rates comparable across groups?
• Are predicted probabilities calibrated within groups?
• Is access to the process equitable?
• Does the system produce a justified outcome, not merely a statistically balanced one?

Several fairness criteria can conflict, especially when groups have different base rates. Therefore, the analyst must document the normative choice, affected groups, trade-offs and decision authority. Fairness assessment should include intersectional groups where sample sizes permit, confidence intervals or uncertainty, qualitative review and examination of real-world impacts.

Mitigation can occur before modelling (better sampling and measurement), during modelling (constraints or reweighting) or after modelling (threshold and workflow changes). Technical mitigation cannot substitute for institutional reform. An appeal process, human review and correction of source records may be more effective than algorithmic adjustment alone.

7. Human Oversight and Appropriate Automation

Human involvement is meaningful only when the person has adequate information, time, authority, competence and independence. A nominal human reviewer may simply accept every recommendation, a phenomenon known as automation bias. Conversely, excessive warnings may cause alert fatigue.

Useful patterns include:

• Human-in-the-loop: a person approves or rejects each consequential action.
• Human-on-the-loop: the system acts within defined limits while a person monitors and can intervene.
• Human-in-command: an authorized person can stop the system and remains responsible for overall operation.
• Abstention or escalation: the model declines to decide when confidence, data quality or case complexity is inadequate.

Automation should be proportionate to risk. Low-risk recommendations may be automated with sampling audits. High-impact decisions generally require documented human judgment, reasons, appeal and the possibility of reversal. The system should show uncertainty rather than presenting a probability as a fact. Users should be trained not only in operation but also in limitations and challenge procedures.

8. Monitoring and Control

Monitoring must cover the whole system, not only model accuracy. Useful indicators include:

• Input drift: changes in the distribution of incoming features.
• Concept drift: changes in the relationship between inputs and outcomes.
• Performance drift: deterioration in precision, recall, calibration or business outcomes.
• Fairness drift: changing error or outcome differences between groups.
• Operational health: latency, availability, failures and queue length.
• Security and privacy: unauthorized access, unusual queries and data exposure.
• Human factors: override rates, complaints, appeals and automation-bias indicators.

A monitoring design should specify metric definitions, data sources, frequency, thresholds, responsible owners and actions. Alerts without response procedures are not effective controls. A mature operating model includes incident classification, investigation, rollback, communication, remediation and post-incident learning.

A useful governance separation distinguishes model development from independent approval and operational monitoring. Changes to training data, features, thresholds, prompts or external tools should be versioned and reviewed according to risk. Production logs should support audit without retaining unnecessary personal information.

9. Responsible AI Governance

Responsible AI combines technical controls with organizational accountability. A governance framework commonly includes:

1. Purpose limitation: use the system only for documented purposes.
2. Data governance: lawful collection, minimization, quality, provenance, retention and access control.
3. Risk classification: stronger review for systems with greater potential harm.
4. Transparency: clear communication about automation, data use and limitations.
5. Human accountability: named owners, review authority and appeal routes.
6. Security and robustness: protection against misuse, manipulation and failure.
7. Continuous assurance: testing, monitoring, audit and incident response.

Privacy principles are especially important because AI systems can infer sensitive attributes even when those attributes were not explicitly collected. Analysts should assess purpose, legal basis where applicable, consent or notice requirements, retention, access and the rights of individuals. Sensitive data should not be copied into development environments merely for convenience.

Regulatory obligations vary by jurisdiction and sector. A system analyst must consult applicable privacy, equality, consumer-protection, employment, health, financial and AI-specific requirements rather than assuming that a general ethical statement is sufficient. Standards and risk-management frameworks can provide structure, but they do not remove the need for domain-specific legal advice.

10. Applied Case: University Student-Success Support System

Background

The following is an illustrative case. A university wishes to identify students who may benefit from timely academic support. It proposes a predictive system using attendance, assessment submissions, course engagement and requests for assistance.

Problem

The initial proposal is to produce a “dropout risk score” and automatically send high-risk students to disciplinary staff. This framing creates risks: historical withdrawal data may reflect financial hardship or unequal support; students may be stigmatized; staff may treat a prediction as a fact; and the system may use information collected for purposes unrelated to support.

Analysis

The analyst reframes the purpose as: “Offer voluntary, timely and appropriate support to students who show signs of difficulty.” The system is not authorized to punish, restrict enrolment or make final judgments about a student. Stakeholders include students, advisers, instructors, disability-support staff, data-protection officers and institutional leaders.

Requirements include:

• use only data available before the support contact;
• display contributing signals and data freshness to advisers;
• allow students to correct inaccurate records;
• provide an opt-out or clear notice where required;
• require adviser review before contact;
• record the support offered and the student’s response;
• evaluate performance and outcomes across relevant groups;
• prohibit use for disciplinary or admissions decisions;
• delete or restrict retention of unnecessary predictions.

Approach and solution

The team first tests a transparent baseline based on missed submissions and repeated non-attendance, then compares it with a more complex model. Evaluation emphasizes recall for students who request support, precision to avoid overwhelming advisers, calibration and subgroup outcomes. The model can abstain when required fields are missing or the case is outside the training population.

The interface presents a support-oriented message rather than a label such as “likely dropout.” Advisers see recent signals, limitations and suggested questions, not an automatic conclusion. Students receive information about the purpose of the system and a route to challenge inaccurate data. A governance panel approves the purpose, reviews subgroup results and receives monthly incident reports.

Outcome

In the pilot, the university finds that a simple model performs nearly as well as a complex model and is easier for advisers to understand. Some students are incorrectly flagged because a temporary illness caused missed assessments. Human review prevents automatic escalation, while a referral pathway connects students to academic, financial and wellbeing services. Monitoring shows that students with limited digital access are underrepresented in engagement data, leading the university to add non-digital support channels rather than merely changing the algorithm.

Learning points

The case demonstrates that the central design question is not “Which model is most accurate?” It is “What support decision is justified, for whom, under what safeguards?” A modest model embedded in a respectful process can be more responsible and useful than a sophisticated model connected to a punitive workflow.

11. Best Practices for System Analysts

1. Define the decision, affected people and acceptable use before selecting technology.
2. Establish a data dictionary, lineage and time boundary for every important variable.
3. Use a transparent baseline and compare complexity against operational value.
4. Test data quality, leakage, subgroup performance and realistic deployment conditions.
5. Treat explanations, uncertainty and appeals as requirements, not optional features.
6. Design meaningful human oversight with authority, training and escalation.
7. Monitor data, model, workflow, fairness, security and outcomes together.
8. Version models, datasets, prompts, thresholds and policies.
9. Minimize personal data and restrict access throughout the lifecycle.
10. Plan rollback, incident response and retirement before deployment.
11. Involve domain experts and affected communities in evaluation.
12. Prefer a simpler or non-AI solution when it meets the need with less risk.

Challenges and Limitations

AI systems face irreducible uncertainty, incomplete data and changing environments. Fairness metrics may conflict; explanations may simplify complex evidence; monitoring outcomes can be delayed; and human reviewers may introduce their own inconsistency. Privacy-preserving measures can reduce analytical detail, while extensive data collection can increase surveillance risk. Generative AI introduces additional concerns, including fabricated content, prompt injection, unauthorized disclosure and unstable outputs.

No governance framework guarantees a beneficial result. Responsible design requires judgment, institutional accountability and willingness to stop or redesign a system. System analysts should communicate uncertainty honestly and resist claims that a technical score is objective merely because it is calculated automatically.

Contemporary Developments

Current practice increasingly emphasizes machine-learning operations (MLOps), model registries, automated testing, data contracts, privacy-enhancing technologies and continuous evaluation. Generative AI systems combine foundation models with retrieval, tools and workflow controls; their requirements therefore include source attribution, output validation and restrictions on autonomous actions. Synthetic data can support testing but may reproduce the biases of the data-generating process and should not be treated as automatically private or representative.

The field is also moving from model-centric evaluation toward system-level evaluation. This includes measuring whether the intervention improves outcomes, whether users understand it, whether affected people can contest it and whether the organization can detect and correct harm. These developments reinforce the central lesson of system analysis: intelligent behaviour emerges from the interaction of technology and its environment.

Chapter Summary

Analytics and AI systems should be analysed as socio-technical decision systems rather than isolated algorithms. Effective design begins by defining the decision, stakeholders, impacts and acceptable use. Data pipelines require source governance, quality controls, lineage and protection against leakage. The model lifecycle includes framing, data and label design, development, validation, deployment, monitoring and retirement.

Performance measures such as accuracy, precision and recall answer different questions and must be interpreted in context. Explainability should be faithful and stakeholder-appropriate. Bias can enter at every lifecycle stage, and fairness requires explicit normative choices rather than a single universal metric. Human oversight must provide genuine authority, information and escalation. Responsible operation requires monitoring of data, performance, fairness, workflow, security and outcomes, supported by documentation, accountability, privacy and incident response. The illustrative student-support case shows why purpose, safeguards and institutional context matter as much as model performance.

Key Terms

Analytics: Systematic examination of data to produce descriptions, insights, predictions or recommendations.

Artificial intelligence: Computational capability associated with tasks such as perception, reasoning, language or learning.

Machine learning: Methods that learn patterns or relationships from data.

Decision support: Information or recommendations intended to assist a human or organizational decision.

Data pipeline: The sequence through which data are collected, moved, transformed, stored and used.

Data lineage: A record of data origin, transformations and uses.

Data leakage: Use of information during development that would not be available at the real decision time.

Feature: An input variable used by an analytical or machine-learning model.

Label: The target outcome used for supervised learning or evaluation.

Model lifecycle: The stages from problem framing and development through operation and retirement.

Precision: The proportion of predicted positive cases that are actually positive.

Recall: The proportion of actual positive cases detected by a system.

Calibration: Agreement between predicted probabilities and observed frequencies.

Explainability: Communication of how or why a system produced an output.

Bias: Systematic distortion that can produce inaccurate or inequitable results.

Human-in-the-loop: A design in which a person participates in each consequential decision.

Drift: Change in input data, relationships or performance over time.

Automation bias: The tendency to over-rely on an automated recommendation.

Responsible AI: Development and operation that address safety, fairness, privacy, accountability and social impact.

Model governance: Policies and controls for approving, documenting, monitoring and changing models.

Review Questions

Short Answer Questions

1. Distinguish descriptive, predictive and prescriptive analytics.
2. What is data leakage, and why can it produce misleading evaluation results?
3. Define precision and recall in a binary classification system.
4. Why is human involvement alone insufficient to guarantee responsible automation?
5. Name four dimensions that should be included in monitoring an AI system.

Long Answer Questions

1. Explain how a system analyst should convert a vague request for an AI solution into testable functional and non-functional requirements.
2. Describe the model lifecycle and identify the principal risks at each stage.
3. Discuss why fairness cannot be reduced to one universal statistical metric.
4. Explain the role of explainability for developers, operational users, affected persons and auditors.
5. Design a governance approach for a high-impact predictive system, including documentation, approval, monitoring and incident response.

Analytical / Application Questions

1. A fraud detector produces TP = 120, TN = 7,800, FP = 180 and FN = 100. Calculate accuracy, precision and recall, and explain why accuracy alone may be inadequate.
2. A university wants to use historical disciplinary records to predict future misconduct. Analyse at least five risks in the proposed target, data and workflow, and recommend a safer alternative purpose.
3. Design a human-oversight and monitoring plan for an AI system that recommends medical appointment prioritization. Specify when it must abstain, what users should see and which outcomes should be audited.

Multiple Choice Questions with Answers

Question 1

Which analytics type primarily asks, “What should be done?”

A. Descriptive analytics

B. Diagnostic analytics

C. Predictive analytics

D. Prescriptive analytics

Correct Answer: D. Prescriptive analytics

Question 2

Which practice most directly prevents data leakage?

A. Using the largest possible feature set

B. Ensuring every feature was available at the real decision time

C. Removing all missing records

D. Increasing the model’s number of parameters

Correct Answer: B. Ensuring every feature was available at the real decision time

Question 3

A classifier has TP = 80, TN = 850, FP = 50 and FN = 20. What is its recall?

A. 61.5%

B. 80%

C. 85%

D. 93%

Correct Answer: B. 80%

Question 4

What does precision measure?

A. The proportion of actual positives detected

B. The proportion of all cases classified correctly

C. The proportion of predicted positives that are truly positive

D. The proportion of actual negatives correctly rejected

Correct Answer: C. The proportion of predicted positives that are truly positive

Question 5

Which statement best describes meaningful human oversight?

A. A person is present but must accept every model output

B. A person has information, authority, competence and time to intervene

C. A person reviews the system only after it is retired

D. A person receives no explanation to preserve objectivity

Correct Answer: B. A person has information, authority, competence and time to intervene

Question 6

Why might excluding a protected attribute fail to eliminate bias?

A. Models cannot operate without protected attributes

B. Other variables may act as proxies for it

C. Protected attributes are always irrelevant

D. Removing the attribute guarantees equal outcomes

Correct Answer: B. Other variables may act as proxies for it

Question 7

Which is an example of concept drift?

A. A server becomes unavailable

B. A feature name is changed in documentation

C. The relationship between inputs and outcomes changes over time

D. A user forgets a password

Correct Answer: C. The relationship between inputs and outcomes changes over time

Question 8

Which item is most appropriate for a model card or equivalent model documentation?

A. Only the model’s marketing slogan

B. Purpose, limitations, evaluation results and intended use

C. An unversioned copy of the production database

D. A guarantee that the model cannot fail

Correct Answer: B. Purpose, limitations, evaluation results and intended use

Question 9

What is the safest general approach when a high-impact model encounters poor-quality or insufficient data?

A. Automatically make the decision using a random value

B. Hide the uncertainty from the user

C. Abstain or escalate according to a defined procedure

D. Increase the decision threshold without review

Correct Answer: C. Abstain or escalate according to a defined procedure

Question 10

Which governance principle limits an AI system to its documented and authorized purpose?

A. Purpose limitation

B. Feature expansion

C. Unrestricted reuse

D. Model opacity

Correct Answer: A. Purpose limitation