Observability, Site Reliability and Disaster Recovery

Introduction

Production AI systems are not complete when a model achieves high accuracy in a notebook. They must remain available, responsive, correct, secure, reproducible, and recoverable while serving real users and processing changing data. Cloud infrastructure makes it possible to scale training and inference, but it also introduces distributed dependencies: model servers, feature stores, data pipelines, object storage, databases, message queues, identity services, GPUs, networks, and third-party APIs. A failure in any of these components can affect the user experience or silently reduce prediction quality.

This chapter presents the operational foundations required to run production AI systems. It examines observability through metrics, logs, and traces; service-level indicators and objectives; incident response; reliability engineering; chaos testing; backup and high availability; and disaster recovery. The emphasis is not merely on collecting data, but on using evidence to diagnose failures and make informed operational decisions.

Learning Objectives

After studying this chapter, students should be able to:

1. Explain the roles of metrics, logs, traces, and telemetry in cloud AI operations.

2. Design useful service-level indicators and service-level objectives for AI services.

3. Diagnose performance and reliability failures using a systematic approach.

4. Describe incident response, post-incident learning, and error-budget practices.

5. Compare redundancy, high availability, backup, and disaster recovery.

6. Evaluate recovery point objective and recovery time objective requirements.

7. Design a basic disaster-recovery plan for a production AI workload.

8. Assess the benefits and risks of chaos testing and automated remediation.

1. Reliability as a Property of the Whole AI System

Reliability is the probability that a system performs its required function under stated conditions for a stated period. In AI, the required function includes more than returning an HTTP response. A prediction service may need to return a valid prediction within a latency target, use an appropriate model version, obtain fresh features, preserve data confidentiality, and produce an auditable result.

A useful distinction is between technical availability and effective availability. A service can be technically reachable while producing stale features, malformed outputs, or predictions with unacceptable latency. Effective availability therefore includes the dimensions that matter to users and business processes.

AI systems have failure modes that differ from conventional web applications:

• Data-quality failure: missing, delayed, duplicated, or incorrectly encoded input data.
• Concept drift: the relationship between inputs and outcomes changes over time.
• Model failure: a model artifact is incompatible with its runtime or has degraded accuracy.
• Resource failure: GPUs, memory, storage, or network capacity is exhausted.
• Dependency failure: a feature store, identity provider, or external API becomes unavailable.
• Pipeline failure: training, validation, deployment, or feature-generation jobs fail.
• Silent failure: the service remains available but produces systematically poor outputs.

Operations must therefore monitor both system health and AI-specific health.

2. Observability: From Telemetry to Explanation

Monitoring asks whether known conditions are present. Observability asks whether internal system states can be inferred from externally visible outputs. In practice, observability is achieved by instrumenting services and correlating telemetry.

2.1 Metrics

A metric is a numerical measurement recorded over time. Common categories include:

• Counter: a value that generally increases, such as requests received or errors produced.
• Gauge: a value that can rise or fall, such as GPU memory utilization or queue depth.
• Histogram: a distribution of observations, such as request latency.
• Summary: a client- or library-computed statistical summary of observations.

Useful AI-service metrics include request rate, error rate, latency percentiles, batch size, queue delay, model-loading time, GPU utilization, GPU memory, feature freshness, prediction distribution, and the proportion of rejected or missing inputs.

Averages alone are dangerous. If 99 requests take 20 ms and one takes 10 seconds, the average is approximately 119.8 ms, while the user experiencing the slow request sees 10 seconds. Percentiles such as p95 and p99 reveal tail latency, although percentiles must be interpreted carefully across aggregation windows and service instances.

Metric labels should be selected carefully. A label such as model version is often useful; a label containing user ID or an unrestricted request ID can create high cardinality, excessive cost, and privacy risk. Aggregation should preserve diagnostic value without recording unnecessary personal data.

2.2 Logs

A log is a time-stamped record of an event. Structured logs, usually encoded as JSON, are more useful than unstructured text because fields can be filtered and aggregated. A production inference log might contain a correlation ID, service name, model version, region, status code, duration, and error class.

Logs should support questions such as:

• Which model version generated the response?
• Did failure occur before or after feature retrieval?
• Which dependency returned an error?
• How many requests were retried?
• What configuration was active at the time?

Logs should not contain passwords, access tokens, raw sensitive records, or unrestricted personal information. Redaction, access control, retention policies, and encryption are essential. Excessive logging can also increase cost and expose sensitive data.

2.3 Traces

A distributed trace follows one request across services. It consists of spans representing operations such as API authentication, feature retrieval, model inference, and response serialization. Trace context propagation allows an operator to connect a front-end request to downstream work.

Tracing is particularly valuable for latency decomposition. If total request latency rises, traces can show whether the increase comes from queueing, network calls, database access, model computation, or serialization. Sampling is commonly used to control overhead. During an incident, systems may increase sampling for errors or unusually slow requests.

2.4 Correlation and the Observability Pipeline

Metrics identify that a problem exists; logs provide event detail; traces show causal paths. A practical observability platform correlates these signals using a shared trace or request identifier and consistent timestamps. Telemetry collection must itself be reliable: agents need buffering, secure transport, access control, and a defined policy for what happens when the observability backend is unavailable.

A useful operational dashboard should present service-level indicators first, then dependency health, resource metrics, and AI-specific quality indicators. Dashboards should support action rather than display every available measurement.

3. Service-Level Indicators and Objectives

A service-level indicator (SLI) is a quantitative measure of a service aspect important to users. A service-level objective (SLO) is a target value or range for an SLI over a stated period. A service-level agreement (SLA) is a formal commitment, often including consequences if targets are missed. An SLO is an engineering target; an SLA may be a contractual obligation.

3.1 Choosing SLIs

Common SLIs include:

• Availability: successful valid requests divided by eligible requests.
• Latency: the proportion of valid requests completed below a threshold.
• Freshness: the proportion of features or data updated within an allowed age.
• Correctness: the proportion of outputs meeting validation rules or verified quality criteria.
• Throughput: completed work per unit time.
• Durability: the proportion of stored objects that remain retrievable without loss.

The numerator and denominator must be explicitly defined. For example, if availability excludes all requests rejected because of capacity, the reported value can be misleading. Eligibility rules should be stable, documented, and resistant to manipulation.

3.2 Error Budgets

For an availability SLO of 99.9% over a 30-day month, the permitted unavailability is 0.1% of the period. Using a 30-day approximation, the total period is 43,200 minutes, so the error budget is 43.2 minutes. This budget represents the amount of unreliability that can be tolerated before the objective is missed.

Error budgets create a decision framework. If the budget is being consumed rapidly, teams may pause risky deployments and prioritize reliability work. If substantial budget remains, the organization may accept carefully controlled changes. The budget is not permission to cause failures; it is a quantitative way to balance innovation and reliability.

3.3 AI-Specific Objectives

Traditional availability objectives are insufficient for many AI services. Additional objectives may specify that:

• 99% of online features are no older than five minutes.
• 99.5% of requests return a schema-valid prediction.
• model quality on a monitored reference set remains above a defined threshold.
• explanations or audit records are generated for regulated decisions.
• training jobs complete before a scheduled decision cycle.

Accuracy objectives require careful statistical design. A measured accuracy value depends on the evaluation sample, labeling quality, population, and confidence in the estimate. It should not be interpreted as a permanent property of the model.

4. Diagnosing Performance and Reliability Failures

Effective diagnosis begins by defining the symptom precisely. “The system is slow” should become a statement such as “p99 inference latency in the European region increased from 400 ms to 2.1 seconds after deployment version 8.4.” The operator then establishes scope, start time, affected users, and recent changes.

A practical diagnostic sequence is:

1. Confirm the signal. Check the SLI, alert validity, time window, and affected region.

2. Assess impact. Determine whether the issue affects availability, latency, correctness, privacy, or data freshness.

3. Check recent changes. Review deployments, configuration, feature schemas, dependency versions, and infrastructure events.

4. Decompose latency and errors. Use traces, dependency metrics, queue depth, and resource utilization.

5. Compare cohorts. Analyze model version, region, tenant, device type, input size, and traffic class.

6. Form and test hypotheses. Change one relevant variable where possible and use controlled rollback or traffic shifting.

7. Mitigate first. Restore service through rollback, scaling, failover, rate limiting, or disabling a nonessential feature.

8. Investigate root causes after stabilization. Preserve evidence and document the timeline.

A causal explanation should distinguish proximate cause from contributing conditions. For example, a GPU out-of-memory error may be the proximate cause, while an unbounded input length, missing load test, and inadequate alerting are contributing conditions.

Worked Example: Error Budget and Incident Impact

Suppose an inference API has a 99.9% monthly availability SLO and uses a 30-day month. The error budget is 43.2 minutes. An incident causes complete unavailability for 18 minutes, followed by 30 minutes in which 20% of requests fail.

The full outage consumes 18 minutes. The partial outage consumes the equivalent of 30 × 0.20 = 6 minutes of failed service. Total budget consumption is therefore 24 equivalent minutes, leaving 43.2 − 24 = 19.2 equivalent minutes. This calculation assumes that the SLI counts failed requests proportionally and that the stated periods are within the same SLO window. It does not imply that users experienced only 24 minutes of inconvenience; partial failures may affect particular users disproportionately.

5. Incident Response and Post-Incident Learning

An incident is an event that threatens or degrades service quality. Incident response is a coordinated process for detection, triage, communication, mitigation, recovery, and learning.

Important roles include:

• Incident commander: coordinates decisions and priorities.
• Operations or technical lead: directs diagnosis and mitigation.
• Communications lead: informs stakeholders and customers.
• Scribe: records events, decisions, commands, and timestamps.
• Subject-matter experts: advise on models, data, infrastructure, or security.

A response process commonly includes detection, severity classification, declaration, containment, mitigation, recovery, validation, and closure. Severity should reflect user and business impact, not merely technical complexity. A security or privacy incident may require a separate escalation path and legal notification.

Runbooks should state symptoms, checks, safe commands, rollback steps, escalation contacts, and validation criteria. Automation is valuable for repetitive actions such as scaling or rollback, but irreversible actions should require appropriate authorization.

A blameless post-incident review focuses on system conditions and decisions made with the information available at the time. It should produce concrete actions with owners and deadlines. Useful outputs include improved tests, safer deployment controls, better alerts, capacity changes, revised runbooks, and clearer ownership. “Human error” is rarely a sufficient root cause because reliable systems anticipate and constrain predictable mistakes.

6. High Availability, Backup, and Disaster Recovery

These concepts are related but not interchangeable.

• High availability (HA) reduces service interruption during component failures through redundancy, health checks, failover, and elimination of single points of failure.
• Backup creates recoverable copies of data or configuration.
• Disaster recovery (DR) is the people, processes, technology, and facilities used to restore operations after a major disruption.
• Business continuity includes broader organizational capabilities for continuing critical functions.

A replicated database may provide HA but does not replace backups. If corrupted data is replicated immediately, every replica may contain the corruption. Backups must be versioned, protected from accidental deletion, and tested through restoration.

6.1 Availability Patterns

Common patterns include:

• Active-passive: one environment serves traffic while a standby is ready to assume service.
• Active-active: multiple environments serve traffic concurrently.
• Multi-zone deployment: replicas are distributed across independent availability zones.
• Multi-region deployment: services operate in geographically separated regions.
• Stateless serving: request-serving instances keep minimal local state, allowing replacement and scaling.

Active-active systems can reduce failover time but create challenges in data consistency, conflict resolution, routing, cost, and operational complexity. Multi-region deployment is not automatically safer if configuration, identity, or data dependencies remain centralized.

6.2 Recovery Objectives

The recovery point objective (RPO) is the maximum tolerable amount of data loss measured in time. An RPO of 15 minutes means that, after recovery, the organization accepts losing at most approximately the most recent 15 minutes of changes under the recovery design.

The recovery time objective (RTO) is the target time to restore an acceptable level of service after disruption. RTO includes detection, decision-making, provisioning, restoration, validation, and traffic migration—not merely the time to copy data.

For an online model service, the model artifact, feature definitions, preprocessing code, container image, dependency versions, secrets-management configuration, and routing configuration may all be required for recovery. Backing up only the model file is insufficient.

6.3 Backup Design

A robust backup strategy addresses:

• frequency and retention periods;
• full, incremental, or snapshot methods;
• encryption in transit and at rest;
• access control and separation of duties;
• immutable or isolated copies to resist ransomware;
• cross-zone or cross-region storage;
• consistency of database and object-storage snapshots;
• restoration testing and evidence.

The 3-2-1 principle is a useful baseline: maintain at least three copies, on at least two types of storage or media, with at least one copy separated from the primary environment. It is a guideline rather than a universal guarantee; regulated or mission-critical systems may require stronger controls.

7. Disaster-Recovery Planning for AI Workloads

A DR plan begins with business impact analysis. The organization identifies critical services, dependencies, acceptable data loss, maximum interruption, regulatory requirements, and recovery priorities. Workloads are then assigned recovery tiers.

A plan should specify:

1. systems and data in scope;

2. owners and escalation paths;

3. RTO and RPO for each service;

4. backup and replication mechanisms;

5. infrastructure-as-code or reproducible environment definitions;

6. model and dataset versioning;

7. recovery procedures and decision authority;

8. security controls and key recovery;

9. validation tests;

10. communication and regulatory procedures.

AI-specific recovery requires preserving lineage. Teams should be able to identify which training data, code, hyperparameters, feature transformations, model artifact, and serving configuration produced a deployed model. If the latest model is defective, the system should support rollback to a known-good model with compatible schemas.

Recovery should be staged. First restore identity, networking, and storage access; then data services and feature generation; then model serving; finally noncritical analytics and experimentation. Smoke tests should verify authentication, representative predictions, latency, feature freshness, logging, and auditability before full traffic is restored.

8. Chaos Engineering and Resilience Testing

Chaos engineering is the disciplined practice of conducting controlled experiments to learn how a system behaves under adverse conditions. Examples include terminating instances, introducing network latency, denying access to a dependency, exhausting a test resource, or serving stale feature data in a controlled environment.

A sound experiment has a hypothesis, defined scope, safeguards, observability, abort conditions, and an expected steady state. For example: “If one inference-serving instance in a zone is terminated, successful requests will remain above the availability SLO because traffic is redistributed within 60 seconds.” The experiment should begin in a nonproduction environment or with a small production blast radius, unless mature controls justify otherwise.

Chaos testing differs from uncontrolled failure. It should not be performed during an unresolved incident, without authorization, or without a way to stop the experiment. Results should lead to engineering changes, not merely a report.

Resilience tests include backup restoration, region failover, dependency outage simulation, queue recovery, model rollback, and loss of observability. A disaster-recovery exercise is successful only when the required service is restored and validated within its RTO and with data loss within its RPO.

9. Governance, Ethics, and Security

Operational telemetry can contain sensitive information. Data minimization, purpose limitation, retention controls, encryption, role-based access, and audit logs are therefore essential. Observability systems should not become an uncontrolled secondary data warehouse.

AI incidents can harm groups unevenly. Monitoring should examine performance across relevant demographic or operational cohorts where lawful and ethically justified. A system may meet overall accuracy objectives while failing a vulnerable subgroup. Alert thresholds and incident procedures should account for these risks.

Regulated decisions may require reproducibility, explanation, human oversight, retention of decision records, and notification procedures. Model rollback must consider whether the previous version remains legally or operationally appropriate. Security incidents involving poisoned data, compromised models, or exposed prompts require coordination among reliability, security, privacy, and governance teams.

10. Contemporary Developments and Professional Practice

Modern cloud platforms increasingly provide managed telemetry, distributed tracing, automated scaling, policy-as-code, and managed model registries. OpenTelemetry has encouraged common instrumentation approaches across metrics, logs, and traces. Service meshes can provide traffic policy and telemetry, although they also add operational complexity.

AI-assisted operations can summarize alerts, correlate events, suggest probable causes, and draft incident timelines. Such tools should support—not replace—human judgment. Their outputs may be wrong, may expose sensitive telemetry, and should not execute high-impact remediation without controls.

Current professional practice emphasizes reliability as a product property, deployment safety through canary or blue-green releases, reproducible infrastructure, continuous validation of data and models, and explicit ownership of every critical dependency. Reliability engineering is most effective when introduced during system design rather than after the first major incident.

Illustrative Case: Regional Failure of a Fraud-Detection Service

Background

A fictional financial technology company operates a cloud-based fraud-detection API. Transactions are scored using a model server, a feature store, a streaming feature pipeline, and a managed database. The service has a 99.95% monthly availability SLO and an RTO of 30 minutes. Features may be no more than ten minutes old, and the latest model and previous approved model are stored in a registry.

Problem

A cloud-region networking problem causes intermittent access to the feature store. HTTP availability falls, p99 latency rises, and feature freshness begins to exceed ten minutes. The model server remains healthy, so a CPU dashboard alone does not reveal the full impact.

Analysis

The availability SLI detects failed requests. Traces show that most slow requests spend time waiting for feature retrieval. Logs correlate failures with the affected region and dependency error codes. A feature-freshness metric reveals that the streaming pipeline is also delayed. The incident commander classifies the event as high severity because both service performance and fraud-detection quality are affected.

Approach and Solution

The team stops a planned deployment, shifts traffic to a second region, and enables a controlled fallback that uses a cached feature set only for transactions eligible under the risk policy. Requests requiring fresh features are queued or reviewed by an alternative process. The team validates model version, schema compatibility, authentication, latency, and audit logging in the second region before increasing traffic. After recovery, it replays affected events and reconciles decisions.

Outcome

Traffic is restored within the 30-minute RTO. The incident consumes part of the error budget, and the post-incident review identifies insufficient dependency-specific alerts and incomplete cross-region pipeline testing. Actions include independent feature-pipeline replication, a freshness-based traffic policy, quarterly failover exercises, and a runbook for safe fallback decisions.

Learning Points

The case demonstrates that model-server health does not establish end-to-end health. Effective observability must include dependencies and AI-specific indicators. A fallback is safe only when its limitations are understood and governed. HA, observability, incident response, and DR must be designed as one operational system.

Challenges and Limitations

Observability has cost, performance, privacy, and complexity trade-offs. Excessive telemetry can obscure important signals. Percentiles can be misleading when calculated incorrectly across distributed instances. Correlation does not prove causation, and automated anomaly detection can generate alert fatigue.

SLOs may encourage optimization of measurable indicators while neglecting unmeasured quality. Accuracy and fairness metrics may be delayed because labels arrive later. DR plans can become obsolete when architectures change. Backups can appear successful while restoration is impossible. Chaos experiments may provide false confidence if their scenarios are too narrow.

These limitations require periodic review, realistic testing, clear ownership, and engineering judgment. Reliability is not achieved by a dashboard or a contractual number; it emerges from architecture, process, people, and continuous learning.

Best-Practice Checklist

• Define user-centered SLIs and document their numerator and denominator.
• Monitor availability, latency, errors, freshness, correctness, and resource health.
• Use structured logs, trace correlation, careful sampling, and privacy controls.
• Alert on actionable symptoms rather than every unusual measurement.
• Maintain tested runbooks and clearly assigned incident roles.
• Use error budgets to balance delivery speed and reliability investment.
• Deploy across failure domains and remove single points of failure.
• Back up data, configurations, model artifacts, and recovery dependencies.
• Test restoration and failover against stated RTO and RPO values.
• Version models, data, code, schemas, and infrastructure.
• Conduct controlled chaos and disaster-recovery exercises.
• Review incidents blamelessly and track corrective actions to completion.

Chapter Summary

Production AI reliability requires end-to-end attention to infrastructure, data, models, dependencies, and users. Observability combines metrics, logs, and traces to make system behavior explainable. SLIs measure user-relevant service properties, while SLOs define targets and error budgets support informed trade-offs. Effective incident response emphasizes rapid impact assessment, mitigation, communication, and blameless learning.

High availability reduces interruption through redundancy and failover, but it does not replace backups or disaster recovery. RPO defines acceptable data loss, while RTO defines acceptable restoration time. AI recovery additionally requires model, data, feature, code, schema, and configuration lineage. Chaos engineering and restoration exercises test whether theoretical resilience works in practice. Finally, privacy, security, fairness, governance, and human oversight must be integrated into operational design.

Key Terms

Observability: The ability to infer internal system state from externally visible telemetry.

Metric: A numerical measurement recorded over time.

Log: A time-stamped record of an event or state.

Trace: A correlated record of work performed across distributed services.

Span: A timed unit of work within a distributed trace.

Service-Level Indicator (SLI): A quantitative measure of a user-relevant service property.

Service-Level Objective (SLO): A target defined for an SLI over a specified period.

Service-Level Agreement (SLA): A formal service commitment, often contractual.

Error budget: The permitted amount of unreliability under an SLO.

High availability: Design that reduces interruption through redundancy and failover.

Backup: A recoverable copy of data or configuration.

Disaster recovery: Processes and technology for restoring operations after major disruption.

Recovery Point Objective (RPO): Maximum tolerable data loss expressed in time.

Recovery Time Objective (RTO): Target time for restoring acceptable service.

Chaos engineering: Controlled experimentation with failure conditions to test resilience.

Incident commander: Person coordinating an incident response.

Feature freshness: The age or timeliness of data used by a model.

Concept drift: Change in the relationship between inputs and outcomes over time.

Blameless post-incident review: A learning review focused on system conditions rather than individual blame.

Review Questions

Short Answer Questions

1. Distinguish metrics, logs, and traces in an observability system.

2. What is the difference between an SLI and an SLO?

3. Define RPO and RTO.

4. Why does a replicated database not eliminate the need for backups?

5. Name two AI-specific indicators that should be monitored in production.

Long Answer Questions

1. Explain how an operator could use metrics, logs, and traces to diagnose increased inference latency.

2. Describe the roles and stages of an effective incident-response process.

3. Compare active-passive and active-active high-availability architectures.

4. Design a disaster-recovery plan for a cloud model-serving system.

5. Explain how error budgets influence deployment and reliability decisions.

Analytical / Application Questions

1. An API has a 99.9% monthly availability SLO. During a 30-day month, it experiences 12 minutes of complete outage and 40 minutes during which 25% of requests fail. Calculate equivalent budget consumption and determine whether the 43.2-minute error budget is exceeded.

2. A model service is available and returns HTTP 200 responses, but feature freshness exceeds its ten-minute objective and fraud losses increase. Identify the observability gap and propose three corrective actions.

3. Design a controlled chaos experiment to test whether an inference service can tolerate the loss of one availability zone. State the hypothesis, safeguards, telemetry, abort conditions, and success criteria.

Multiple Choice Questions with Answers

Question 1

Which telemetry type is most useful for following one request across an API, feature store, and model server?

A. Gauge

B. Distributed trace

C. Backup snapshot

D. SLA

Correct Answer: B. Distributed trace

Question 2

For a 99.9% availability target over an approximate 30-day month, what is the error budget?

A. 4.32 minutes

B. 43.2 minutes

C. 432 minutes

D. 0.432 minutes

Correct Answer: B. 43.2 minutes

Question 3

Which statement best describes an RPO of 15 minutes?

A. Service must recover within 15 minutes

B. At most approximately 15 minutes of data loss is acceptable

C. Every request must finish within 15 minutes

D. Backups must be retained for 15 minutes

Correct Answer: B. At most approximately 15 minutes of data loss is acceptable

Question 4

Which practice most directly reduces high-cardinality metric problems?

A. Using unrestricted request IDs as metric labels

B. Removing all timestamps from logs

C. Selecting bounded, meaningful label values

D. Recording every input field in a metric

Correct Answer: C. Selecting bounded, meaningful label values

Question 5

What is the primary purpose of an error budget?

A. To calculate cloud storage capacity

B. To balance delivery changes against reliability performance

C. To encrypt model artifacts

D. To replace incident response

Correct Answer: B. To balance delivery changes against reliability performance

Question 6

Which activity is an example of chaos engineering?

A. Deleting production data without safeguards

B. Testing controlled loss of an instance against a stated hypothesis

C. Increasing model training epochs

D. Writing an unstructured application log

Correct Answer: B. Testing controlled loss of an instance against a stated hypothesis

Question 7

Why are model-server health metrics alone insufficient for an AI service?

A. They cannot measure GPU temperature

B. The model server may be healthy while dependencies or feature freshness fail

C. They always measure accuracy incorrectly

D. They automatically disable backups

Correct Answer: B. The model server may be healthy while dependencies or feature freshness fail

Question 8

Which control best protects against a corrupted database being replicated to every replica?

A. Versioned, tested backups

B. Increasing request rate

C. Removing audit logs

D. Using only one availability zone

Correct Answer: A. Versioned, tested backups

Question 9

Who coordinates priorities and decisions during a major incident?

A. Incident commander

B. Database replica

C. Model validator

D. Metric label

Correct Answer: A. Incident commander

Question 10

Which item is especially important for recovering an AI service reproducibly?

A. Only the latest model filename

B. Model, data, code, schema, and serving-configuration lineage

C. Only GPU utilization history

D. Only the number of API requests

Correct Answer: B. Model, data, code, schema, and serving-configuration lineage