7. Machine Learning and Artificial Intelligence at Scale

Introduction and Context

Machine learning (ML) and artificial intelligence (AI) have moved from isolated analytical experiments to core capabilities embedded in commerce, science, government, healthcare, manufacturing, and digital platforms. This transition is not simply a consequence of better algorithms. It reflects the convergence of massive data volumes, distributed computing, specialized hardware, cloud infrastructure, automated data pipelines, and increasingly sophisticated model architectures.

AI at scale refers to the design, training, deployment, monitoring, and governance of intelligent systems whose data, computational, organizational, or societal scope exceeds the capacity of a single-machine or manually managed workflow. Scale may involve billions of records, millions of users, thousands of model versions, real-time predictions, or large foundation models with billions of parameters. It also involves scale of consequence: an apparently small modeling error can affect access to credit, medical treatment, employment, public services, or online information for large populations.

The central problem is therefore not merely how to obtain a high test score. A scalable AI system must solve a chain of interdependent problems: defining a valid task, acquiring and representing data, engineering features, selecting and training models, evaluating generalization, serving predictions efficiently, explaining decisions, resisting manipulation, and maintaining human accountability. A model can be statistically accurate yet operationally unusable, unfair, insecure, irreproducible, or impossible to govern.

This chapter examines that complete perspective. It connects classical supervised and unsupervised learning with deep learning, distributed training, generative and foundation models, model selection, explainability, robustness, and human oversight. The emphasis is doctoral: methods are considered not only for their performance, but also for their assumptions, evidence, limitations, and implications for research and institutional decision-making.

Learning Objectives

By the end of this chapter, the reader should be able to:

1. Distinguish the principal learning paradigms and determine when each is appropriate at large scale.

2. Design data and feature-engineering pipelines that reduce leakage, preserve reproducibility, and support distributed computation.

3. Explain the architecture and trade-offs of distributed model training and large-scale inference.

4. Compare conventional deep models, foundation models, and generative AI systems.

5. Select models using statistical, operational, ethical, and economic criteria rather than accuracy alone.

6. Evaluate explainability, robustness, fairness, uncertainty, and human oversight in high-impact applications.

7. Critically assess risks, governance requirements, and emerging research directions in large-scale AI.

From a Predictive Task to a Scalable Learning System

A machine-learning project begins with a target decision or prediction, not with an algorithm. In supervised learning, observations are paired with labels: a transaction may be labeled fraudulent, a document may belong to a class, or a time series may contain a future demand value. Classification predicts discrete categories; regression predicts continuous values. In unsupervised learning, labels are absent or incomplete, and the objective may be to discover clusters, represent structure, detect anomalies, or reduce dimensionality. Semi-supervised and self-supervised methods exploit large quantities of unlabeled data together with a smaller labeled set or automatically constructed learning signals.

The distinction between a modeling objective and an organizational objective is crucial. A hospital may request a model to predict readmission, but its real objective may be to allocate care resources without worsening disparities. A retailer may seek demand forecasts, while the operational purpose is inventory control under uncertain lead times. These objectives imply different loss functions, decision thresholds, latency requirements, and evaluation designs.

At scale, data are usually distributed across time, geography, business units, or storage systems. The independent and identically distributed assumption is frequently questionable. Distribution shift occurs when the relationship between inputs and outcomes changes; concept drift describes change in the target relationship, while covariate shift describes change in the input distribution. A model trained on historical data may therefore fail not because optimization was inadequate, but because the future is generated by a different process.

Important terminology includes parameters, learned from data; hyperparameters, selected outside the fitting process; an objective or loss function, which quantifies error; an optimizer, which updates parameters; an epoch, representing a pass through training data; and inference, the production-time computation of predictions. Training, validation, and test data must be separated according to the intended deployment setting. In temporal applications, random splitting can leak future information and produce unjustifiably optimistic results.

Feature Engineering in High-Volume Environments

Feature engineering transforms raw events into representations usable by a learning algorithm. Although deep learning can learn representations automatically, engineered features remain important for tabular, temporal, regulated, and resource-constrained applications. Common transformations include normalization, categorical encoding, aggregation, interaction construction, missing-value treatment, text tokenization, and extraction of temporal or spatial characteristics.

The key principle is that a feature must be available at the time a prediction is made. A feature computed using a later outcome, a post-decision status, or a future aggregate creates target leakage. Leakage may remain hidden when data pipelines are assembled from operational databases because timestamps reflect data entry rather than information availability. A robust feature store therefore records event time, ingestion time, transformation logic, lineage, and the point at which a feature becomes valid.

At large scale, feature engineering is a data-systems problem. Batch features may be computed using distributed processing frameworks, whereas low-latency applications require streaming or online computation. The two paths must produce semantically equivalent values; otherwise, training-serving skew arises. Reproducibility requires versioned datasets, code, schemas, transformation environments, and random seeds where meaningful. Data quality checks should address completeness, validity, uniqueness, distributional changes, and referential integrity.

High dimensionality creates both computational and statistical difficulties. Feature selection, regularization, dimensionality reduction, embeddings, and representation learning can reduce noise or cost. However, dimensionality reduction may obscure interpretable variables, and automated selection can amplify historical biases. The appropriate representation is consequently determined by the task, deployment constraints, and accountability requirements, not by compression alone.

Distributed Training and Inference

A single machine becomes inadequate when data, model parameters, or required experimentation exceed its memory or processing capacity. Data parallelism partitions training examples across workers. Each worker computes gradients on a local mini-batch, and gradients are aggregated before parameters are updated. Synchronous training provides consistent updates but may be slowed by the slowest worker, known as the straggler problem. Asynchronous training reduces waiting but can use stale parameters and complicate convergence.

Model parallelism partitions a model across devices when its parameters or intermediate activations cannot fit on one device. Pipeline parallelism divides layers into stages and processes micro-batches through them. Hybrid approaches combine data, tensor, and pipeline parallelism. Distributed training also requires attention to communication topology, parameter synchronization, checkpointing, fault recovery, precision, and reproducibility. Communication can dominate computation, especially when frequent gradient exchanges occur across geographically separated resources.

Large-scale training commonly uses accelerators such as GPUs or specialized tensor-processing hardware. Mixed-precision arithmetic can improve throughput and reduce memory consumption, but it requires numerical safeguards such as loss scaling and monitoring for overflow. Gradient accumulation permits an effectively larger batch when device memory is limited. Larger batches may improve hardware utilization, but they can alter optimization dynamics and generalization; they are not automatically superior.

Inference introduces a different optimization problem. A production service must meet latency, throughput, availability, cost, and privacy requirements. Techniques include batching, caching, quantization, pruning, distillation, compilation, and hardware-aware placement. Edge inference reduces data transmission and latency but may impose severe memory and energy constraints. Model serving should include version control, rollback, shadow evaluation, canary release, and monitoring for both technical and behavioral degradation.

A useful distinction is between scaling up and scaling out. Scaling up increases the resources of a single machine; scaling out distributes work across machines. Scaling out improves capacity but introduces coordination, network, fault-tolerance, and observability costs. Thus, the largest possible architecture is not necessarily the most efficient architecture.

Learning Paradigms and Model Families

Supervised learning remains central where reliable labels exist. Linear and generalized linear models offer strong baselines, transparent coefficients, and efficient training. Tree-based ensembles often perform well on heterogeneous tabular data and can represent nonlinear interactions with limited preprocessing. Neural networks are advantageous where data are abundant and structure is complex, including images, speech, text, and multimodal streams.

Unsupervised learning supports exploratory analysis and representation discovery. Clustering methods such as k-means assume a particular geometry and may produce clusters even when no meaningful groups exist. Density-based methods can identify irregular structures and noise but may be sensitive to scale and local density. Principal component analysis provides a linear low-dimensional representation, while autoencoders learn nonlinear representations. Anomaly detection is especially difficult because rare observations may be errors, novel legitimate behavior, or evidence of attack; validation must therefore involve domain knowledge rather than an arbitrary anomaly score threshold.

Deep learning uses multiple layers to learn hierarchical representations. Convolutional architectures exploit local structure in images and spatial signals. Recurrent architectures model sequential dependence, although attention-based systems have become dominant in many language and multimodal applications. The transformer architecture uses attention to relate elements of a sequence or set and can be trained efficiently through parallel computation. Its computational and memory costs, especially for long contexts, remain important design constraints.

Generative models learn to produce new outputs that resemble a data distribution or satisfy a conditional instruction. Autoregressive language models generate tokens sequentially; diffusion models progressively denoise a sample; variational autoencoders learn probabilistic latent representations; and generative adversarial networks train a generator against a discriminator. Generative quality is multidimensional: fluency or realism does not establish factuality, safety, originality, or usefulness.

Foundation models are broadly pretrained models adapted to many downstream tasks through prompting, fine-tuning, adapters, retrieval augmentation, or tool use. Pretraining can exploit unlabeled data at a scale unavailable to task-specific systems. Yet the resulting model inherits biases, omissions, licensing constraints, and security weaknesses from data and training processes. A foundation model is not automatically reliable in a specialized domain. Domain evaluation, grounding, access control, and human review remain necessary.

Model Selection as Multi-Criteria Reasoning

Model selection should combine predictive performance with operational and institutional requirements. Accuracy may be inadequate for imbalanced classification, where precision, recall, area under a precision-recall curve, calibration, and cost-sensitive measures are more informative. Regression evaluation may use mean absolute error, root mean squared error, quantile loss, or proper scoring rules for probabilistic forecasts. Ranking systems require measures such as precision at a cutoff, recall at a cutoff, or normalized discounted cumulative gain.

Validation must reflect deployment. Nested cross-validation reduces optimistic bias when hyperparameters are selected, but it can be computationally expensive. Time-based validation is appropriate for evolving systems; group-based splitting is needed when observations from the same person, device, or organization are correlated. External validation tests transportability to another population or setting. Confidence intervals, bootstrap analysis, sensitivity analysis, and ablation studies help distinguish robust findings from accidental improvements.

A practical selection framework asks five questions. First, does the model solve the intended decision problem? Second, does it generalize under plausible distribution shifts? Third, can its outputs be calibrated and explained sufficiently for the context? Fourth, can it satisfy latency, cost, privacy, and reliability constraints? Fifth, can the organization monitor, update, audit, and retire it? A smaller model that is stable, interpretable, and inexpensive may be preferable to a marginally more accurate model that cannot be governed.

Explainability, Robustness, and Human Oversight

Explainability concerns the relationship between a model’s behavior and the understanding required by its users, developers, auditors, or affected individuals. Intrinsically interpretable models expose their structure, whereas post hoc methods approximate explanations after prediction. Feature-attribution techniques estimate the contribution of inputs; local surrogate methods approximate behavior around one instance; counterfactual explanations describe changes that might alter an outcome; and global analyses summarize patterns across a population.

Explanations are not synonymous with causes. An attribution may describe a predictive association without showing that changing the feature would change the outcome. Explanations can also be unstable, manipulated, or misleadingly persuasive. Evaluation should therefore examine fidelity, stability, comprehensibility, actionability, and consistency with domain knowledge. In high-impact settings, explanation should complement—not replace—documentation, testing, and appeal mechanisms.

Robustness concerns performance under perturbation, noise, missing information, adversarial manipulation, and distribution shift. Adversarial examples demonstrate that small, deliberately designed input changes can cause incorrect predictions. Robust training, input validation, anomaly detection, ensemble methods, uncertainty estimation, and stress testing can reduce vulnerability, but no single defense is complete. Generative systems introduce additional risks, including prompt injection, data poisoning, sensitive-information disclosure, fabricated content, and insecure tool execution.

Human oversight must be designed as an operating process. A human merely approving automated outputs without time, competence, or authority to disagree provides little protection. Effective oversight defines escalation thresholds, reviewer responsibilities, evidence displays, disagreement recording, override procedures, and periodic audits. Automation bias must be addressed by training and interface design. In some applications, the appropriate decision is not human-in-the-loop prediction but human-led assessment supported by carefully bounded automation.

Governance, Ethics, and Organizational Responsibility

Large-scale AI raises questions of privacy, discrimination, autonomy, intellectual property, labor displacement, safety, and accountability. Data minimization, purpose limitation, access control, retention policies, encryption, and privacy-preserving methods can reduce exposure. Federated learning keeps some data at local sites while sharing model updates, but updates may still leak information and systems remain vulnerable to poisoned participants. Differential privacy provides a formal privacy budget by adding calibrated randomness, at the cost of reduced utility.

Fairness cannot be reduced to one universal metric. Group parity, equalized error rates, calibration, and individual similarity criteria can conflict, particularly when base rates differ. The relevant choice depends on the decision, legal environment, harms, and stakeholder values. Fairness assessment should include intersectional groups, missingness patterns, downstream effects, and the possibility that the target itself reflects unequal institutional treatment.

Responsible governance requires traceability across the model life cycle. Model cards, data documentation, risk assessments, impact assessments, access logs, incident registers, and change-control procedures support accountability. Governance should classify systems according to risk and apply proportionate controls. Procurement is also significant: organizations need contractual rights to audit vendors, understand data use, receive incident notifications, and obtain model and service documentation.

Research ethics require special care when large datasets are scraped, combined, or repurposed. Consent, contextual integrity, representational harm, re-identification risk, and the rights of data subjects must be considered. A model that is technically reproducible may still be ethically inappropriate to reproduce if its data cannot be responsibly shared.

Illustrative Case: A Scalable Fraud Detection Platform

Background

A fictional digital payments company, MeridianPay, processes transactions through mobile applications, online merchants, and international partners. It seeks to reduce financial fraud while limiting false declines that inconvenience legitimate customers.

Situation and Problem

The existing rules engine catches familiar patterns but misses coordinated attacks that change rapidly. A proposal recommends a deep neural classifier trained on several years of transactions. However, fraud labels arrive late, data are highly imbalanced, and legitimate behavior differs by region, product, and customer lifecycle. The company also requires near-real-time decisions and must provide a review pathway for disputed transactions.

Analysis

The project team first defines the decision as risk ranking rather than an unconditional accept-or-reject prediction. It audits label delay, missingness, duplicate accounts, and temporal leakage. Random splitting is rejected because transactions from the same attack campaign can appear in both training and test sets. Instead, the team uses rolling temporal validation and evaluates precision at the review capacity, recall at an operational false-positive limit, calibration, latency, and performance across regions.

Feature engineering combines transaction velocity, device relationships, merchant history, and graph-derived connections. Features are computed using only information available before authorization. A feature registry records event time and transformation versions. Because fraud patterns evolve, the platform includes drift monitoring and scheduled investigation of newly emerging clusters.

Approach and Solution

MeridianPay deploys a layered system: interpretable rules handle known high-confidence patterns; a gradient-boosted model produces a calibrated risk score; and an anomaly detector identifies novel behavior for investigation. A graph representation is used to detect coordinated entities, but it is not allowed to make an irreversible decision without review. High-risk transactions may be blocked, intermediate cases are routed to trained analysts, and low-risk transactions proceed with post-transaction monitoring.

Training is distributed across partitioned historical data, with strict temporal controls and reproducible feature versions. The production service uses a smaller distilled model for low latency while retaining the larger model for offline analysis. Analysts receive reason codes, relevant historical evidence, and an override mechanism. Monitoring tracks drift, subgroup error rates, calibration, reviewer disagreement, model latency, and fraud loss.

Outcome

In this illustrative scenario, the principal improvement is not attributed to a single architecture. It results from aligning labels, validation, feature availability, model specialization, human review, and continuous monitoring. The company accepts that some fraud will remain undetected and treats threshold setting as a business and risk decision rather than a purely technical optimization.

Learning Points

The case demonstrates that scale involves volume, speed, adversarial adaptation, and organizational coordination. It also shows why ensembles of specialized methods may be more governable than one opaque model, and why evaluation must measure operational consequences rather than an isolated accuracy score.

Failure Modes and Research Challenges

Common mistakes include beginning with a fashionable model, optimizing a proxy that does not represent the decision, leaking future information, ignoring label uncertainty, treating imbalanced accuracy as sufficient, and deploying without a rollback plan. Other failures arise when monitoring measures infrastructure uptime but not data drift, fairness, calibration, or user harm. Fine-tuning a foundation model without testing memorization, prompt injection, or domain hallucination is similarly inadequate.

Scaling can also intensify environmental and financial costs. Training and serving large models consume energy and hardware capacity; repeated experimentation multiplies these costs. Efficient data selection, parameter-efficient adaptation, model distillation, sparse computation, and carbon-aware scheduling are active responses, though their benefits depend on measurement boundaries and hardware assumptions.

Research remains open in reliable uncertainty estimation for deep models, continual learning without catastrophic forgetting, causal and counterfactual modeling, privacy-utility trade-offs, evaluation of generative systems, multimodal grounding, and methods for auditing models whose training data and internal representations are difficult to inspect. A doctoral researcher should distinguish benchmark improvement from scientific progress: a new result is meaningful only when the evaluation is sound, the comparison is fair, and the claimed mechanism is supported by evidence.

Best Practices for Scalable AI Projects

A defensible project generally follows these principles:

• Define the decision, affected stakeholders, harm boundaries, and success criteria before selecting a model.
• Establish data lineage, temporal validity, label definitions, and leakage controls.
• Build a simple, interpretable baseline before introducing a complex architecture.
• Separate development, validation, external testing, and production monitoring.
• Treat latency, cost, privacy, fairness, robustness, and maintainability as design variables.
• Use reproducible pipelines, versioned artifacts, documented assumptions, and controlled releases.
• Evaluate uncertainty, subgroup behavior, distribution shift, and plausible adversarial conditions.
• Design meaningful human review, appeals, escalation, and incident response.
• Monitor the complete socio-technical system and retire models when their risk or utility changes.

Current Developments and Emerging Directions

The field is moving toward multimodal foundation models, retrieval-augmented generation, agentic systems that use external tools, small specialized models, and parameter-efficient adaptation. Retrieval can improve grounding by supplying current or domain-specific material, but it does not guarantee that a model will use retrieved evidence correctly. Agents increase capability while expanding attack surfaces and the need for permission controls.

Synthetic data, simulation, federated systems, confidential computing, and privacy-enhancing technologies are also developing rapidly. Synthetic data may help address scarcity, but it can reproduce or amplify source-data bias and may fail to represent rare events. Governance research increasingly emphasizes evaluation standards, provenance, model supply chains, systemic risk, and the accountability of organizations that deploy rather than merely develop AI.

The durable direction is toward integrated AI engineering: data engineering, statistical learning, software reliability, security, ethics, and institutional governance must be designed together. Scale makes these connections unavoidable.

Chapter Summary

Scalable AI is a complete socio-technical system rather than an oversized predictive model. It requires valid problem formulation, leakage-resistant feature engineering, distributed computation, deployment-aware model selection, and continuous evaluation. Supervised, unsupervised, deep, generative, and foundation-model approaches each offer distinctive benefits and assumptions. Distributed training improves capacity but introduces communication and reproducibility challenges; inference requires attention to latency, cost, and reliability. Explainability must be evaluated for fidelity and usefulness, while robustness testing must include shift, adversarial behavior, and uncertainty. Fairness, privacy, documentation, human oversight, and incident response are essential in high-impact settings. The strongest systems align technical performance with operational realities, stakeholder interests, and accountable governance.

Key Terms

Parameter: A value learned by a model during training.

Hyperparameter: A configuration selected outside the parameter-fitting process.

Data leakage: Use of information during training that would not be available at prediction time.

Feature store: A managed system for defining, versioning, and serving model features.

Training-serving skew: A mismatch between feature computation during training and production inference.

Concept drift: Change over time in the relationship between inputs and the target.

Data parallelism: Distribution of training examples across workers that compute coordinated updates.

Model parallelism: Partitioning of a model across computational devices.

Foundation model: A broadly pretrained model adaptable to multiple downstream tasks.

Generative AI: AI that produces new content such as text, images, audio, or code.

Calibration: Agreement between predicted probabilities and observed frequencies.

Distribution shift: A change between the data-generating conditions of training and deployment.

Explainability: The capacity to provide understandable accounts of model behavior or outputs.

Adversarial example: An input deliberately modified to induce an incorrect model response.

Human oversight: Structured human authority to review, challenge, modify, or stop automated action.

Differential privacy: A formal method limiting the information revealed about an individual by an analysis.

Federated learning: Collaborative model training in which data remain at participating sites.

Model drift: Degradation or behavioral change caused by evolving data, relationships, or environments.

Retrieval-augmented generation: Generation grounded by information retrieved from an external collection.

Model card: Documentation describing a model’s intended use, performance, limitations, and risks.

Review Questions

Short Answer Questions

1. Why is random train-test splitting potentially invalid for a time-dependent fraud-detection problem?

2. Distinguish data parallelism from model parallelism.

3. What is training-serving skew, and how can a feature registry reduce it?

4. Why does high predictive accuracy not by itself establish that a foundation model is suitable for a specialized domain?

5. Define calibration and explain its relevance to risk-based decisions.

Long Answer Questions

1. Explain how feature engineering, data lineage, and leakage controls should be designed for a distributed learning system.

2. Compare supervised, unsupervised, self-supervised, deep, and generative learning in terms of data requirements, objectives, and applications.

3. Discuss the architectural and operational trade-offs among data parallelism, model parallelism, synchronous training, and asynchronous training.

4. Develop a multi-criteria framework for selecting a model in a high-impact organizational application.

5. Critically evaluate the roles of explainability, robustness testing, fairness assessment, and human oversight in responsible AI deployment.

Analytical / Application Questions

1. Design an evaluation strategy for a hospital readmission model when patient records are temporally correlated, labels are delayed, and subgroup performance differs.

2. A generative AI assistant produces fluent but unsupported answers and can call enterprise tools. Analyze the risks and propose controls involving retrieval, permissions, monitoring, and human review.

3. Using the MeridianPay case, explain why a layered system may be preferable to a single highly complex model, and identify the evidence needed to justify its deployment.

22. Multiple Choice Questions with Answers

Question 1

Which validation design is most appropriate for a fraud-detection model when attack patterns and labels evolve over time?

A. A random split with identical class proportions

B. A rolling temporal validation design

C. Training accuracy measured on all historical records

D. A split based only on model confidence

Correct Answer: B. A rolling temporal validation design

Question 2

What is the principal purpose of recording feature availability time in a feature store?

A. To increase the number of model parameters

B. To guarantee that all features are normally distributed

C. To prevent information unavailable at prediction time from entering training

D. To eliminate the need for production monitoring

Correct Answer: C. To prevent information unavailable at prediction time from entering training

Question 3

In synchronous data-parallel training, what commonly limits progress when workers have unequal processing speeds?

A. The straggler problem

B. The absence of labels

C. The vanishing of all gradients

D. The inability to use mini-batches

Correct Answer: A. The straggler problem

Question 4

Which statement best distinguishes model parallelism from data parallelism?

A. Model parallelism partitions the model across devices, whereas data parallelism partitions examples

B. Model parallelism is used only for unsupervised learning

C. Data parallelism eliminates communication between workers

D. Data parallelism requires every worker to use different model parameters

Correct Answer: A. Model parallelism partitions the model across devices, whereas data parallelism partitions examples

Question 5

Why might a smaller distilled model be used in production while a larger model is retained offline?

A. Distillation guarantees perfect fairness

B. The smaller model may satisfy latency and resource constraints more effectively

C. The larger model cannot be evaluated offline

D. A distilled model does not require monitoring

Correct Answer: B. The smaller model may satisfy latency and resource constraints more effectively

Question 6

Which claim about post hoc explanations is most accurate?

A. They always establish causal relationships

B. They make fairness testing unnecessary

C. They approximate model behavior and should be evaluated for fidelity and stability

D. They guarantee that a model is robust to adversarial inputs

Correct Answer: C. They approximate model behavior and should be evaluated for fidelity and stability

Question 7

A model has high accuracy but poorly aligned predicted probabilities. Which property is deficient?

A. Calibration

B. Parallelism

C. Tokenization

D. Checkpointing

Correct Answer: A. Calibration

Question 8

Which risk is particularly associated with a generative model that can execute external tools?

A. Prompt injection causing unsafe tool actions

B. The impossibility of producing novel outputs

C. Guaranteed elimination of privacy risk

D. The absence of any need for access controls

Correct Answer: A. Prompt injection causing unsafe tool actions

Question 9

Why can fairness metrics conflict across demographic groups?

A. All fairness metrics measure exactly the same property

B. Different metrics impose different statistical conditions, which may be incompatible

C. Fairness is determined only by model size

D. Calibration automatically satisfies every fairness criterion

Correct Answer: B. Different metrics impose different statistical conditions, which may be incompatible

Question 10

What is the strongest lesson from the MeridianPay illustrative case?

A. A deep neural network should always replace rules

B. Accuracy alone determines the best fraud solution

C. Scalable AI requires alignment among data validity, operational constraints, human review, and monitoring

D. Anomaly detection makes temporal validation unnecessary

Correct Answer: C. Scalable AI requires alignment among data validity, operational constraints, human review, and monitoring