Reliability, Performance and Cloud Operations

Introduction

Cloud systems are not successful merely because they can be deployed. They must remain available, responsive, recoverable, secure, and economically sustainable while requirements, workloads, dependencies, and failure conditions change. Reliability and performance are therefore operational properties of an entire socio-technical system: software, infrastructure, networks, data stores, people, processes, and external providers.

Cloud operations applies engineering discipline to the design and operation of such systems. It combines site reliability engineering (SRE), observability, service-level management, capacity planning, automated scaling, resilience testing, backup, disaster recovery, and business continuity. The central operational question is not whether failure can be eliminated. Failure is inevitable; the question is whether the system can detect, absorb, recover from, and learn from failure within acceptable business limits.

This chapter treats reliability as a measurable and governable engineering objective. It also examines trade-offs. For example, higher availability may increase cost, replication may increase consistency complexity, and aggressive autoscaling may improve latency while increasing expenditure or causing instability.

Learning Objectives

After studying this chapter, students should be able to:

1. Explain SRE principles and distinguish reliability from availability and resilience.

2. Design useful observability using logs, metrics, traces, and events.

3. Define service-level indicators (SLIs), objectives (SLOs), and agreements (SLAs).

4. Apply error budgets to operational and product decisions.

5. Perform basic capacity planning and reason about autoscaling.

6. Design backup, disaster-recovery, and business-continuity strategies.

7. Evaluate chaos-engineering experiments and operational risks.

8. Develop an integrated reliability and operations plan for a cloud service.

1. Reliability as a Cloud Engineering Property

Reliability is the probability that a system performs its required function for a specified period under stated conditions. In operational practice, it includes availability, correctness, durability, recoverability, and acceptable performance. A service may be technically available but unreliable if it returns incorrect results, loses data, or responds too slowly for users.

Availability is the proportion of time that a service is usable. For a simple interval, if total scheduled time is T and unavailable time is U, then:

Availability = (T − U) / T

This formula assumes that the interval and the treatment of planned maintenance have been defined consistently. Availability alone does not measure correctness, latency, or data integrity.

Resilience is the ability to continue providing an acceptable service during disruption and to recover afterward. Reliability is an outcome; resilience is a set of properties and practices that help produce that outcome. Fault tolerance is the ability to continue operation despite particular faults, often through redundancy or isolation. Recoverability concerns restoration after a serious incident.

A useful reliability model distinguishes four states:

• Prevent: reduce the probability of faults through sound design, testing, security, and change control.
• Detect: identify faults and degradation quickly through monitoring and observability.
• Respond: contain impact and restore service through automation and coordinated incident response.
• Learn: remove systemic causes through post-incident analysis and engineering improvement.

Cloud architectures support these goals through multiple availability zones, load balancing, immutable infrastructure, managed services, health checks, queues, circuit breakers, rate limits, replication, and automated deployment rollback. None is universally beneficial. Replication, for example, can improve availability but may introduce replication lag, split-brain risk, cost, and consistency challenges.

2. Site Reliability Engineering

SRE is an engineering approach to operating large-scale services. It treats operations as a software and systems problem, using automation, measurement, service design, and disciplined incident management. SRE does not mean that every organization must copy a particular team structure. Its principles can be adapted to organizational scale and risk.

2.1 Service-level indicators, objectives, and agreements

An SLI is a quantitative measure of a user-relevant aspect of service behavior. Examples include:

• successful request proportion;
• latency of the 95th or 99th percentile of valid requests;
• freshness of a data pipeline;
• successful completion of a batch job;
• durability or loss rate for stored objects.

An SLO is a target value or range for an SLI over a stated time window. For example, “at least 99.9% of valid checkout requests succeed over a rolling 30-day window.” An SLA is a business or contractual commitment, often including remedies if a target is missed. An SLA should not automatically be set equal to an internal SLO: internal targets normally need operating margin before a contractual commitment is breached.

SLIs must have an explicit numerator, denominator, measurement source, time window, and exclusion policy. “The system is reliable” is not an SLI. “The percentage of authenticated API requests returning a valid response within 500 milliseconds, measured at the service edge over 30 days” is much more precise. Exclusions must not be used to conceal failures; they should be justified, documented, and auditable.

2.2 Error budgets

For an availability SLO, the error-budget fraction is:

Error budget = 1 − SLO target

If the target is 99.9% for a 30-day period, the budget is 0.1% of eligible time or requests. For request-based measurement, if 10,000,000 requests are eligible, the budget is 10,000 failed requests. This assumes that the SLI is defined as the proportion of successful eligible events and that failures are counted consistently.

The budget is not permission to cause failure. It is a decision instrument. When the budget is healthy, a team may allocate capacity to feature delivery, controlled experimentation, or modernization. When it is being rapidly consumed, the team should prioritize reliability work, freeze risky changes, or reduce release scope. Error budgets make the trade-off between innovation and operational risk explicit.

An SLO also has a statistical limitation: a monthly aggregate can hide a severe short outage. A service that fails completely for 43.2 minutes in a 30-day month has approximately 99.9% availability, yet users may experience a major incident. Therefore, aggregate SLOs should be complemented by alerting on short windows and by latency, correctness, and saturation indicators.

2.3 Toil and operational maturity

Toil is repetitive, automatable, tactical work that scales with service volume rather than with engineering improvement. Examples include manually restarting failed instances, repeatedly cleaning predictable temporary files, and copying the same incident data into reports. SRE seeks to reduce toil through automation, better defaults, self-healing, and improved system design. Automation must itself be observable and governed; an automated action that amplifies failure can be more dangerous than a manual action.

Operational maturity commonly develops through stages:

1. Reactive operation: incidents are handled manually and knowledge is largely undocumented.

2. Instrumented operation: logs, metrics, alerts, runbooks, and ownership are established.

3. Automated operation: deployment, scaling, recovery, and compliance checks are automated.

4. Adaptive operation: teams use error budgets, predictive capacity models, experimentation, and systematic learning.

3. Observability and Telemetry

Monitoring asks whether a known condition is occurring. Observability asks whether the internal state of a system can be inferred from its external outputs. Observability is especially important in cloud systems, where distributed requests cross services, regions, queues, databases, and provider-managed components.

3.1 Metrics

Metrics are numerical measurements collected over time. Common categories are:

• Availability and correctness: success rate, error rate, rejected requests.
• Latency: average, median, p95, p99, and maximum response time.
• Traffic: requests per second, messages per second, active users.
• Saturation: CPU, memory, connection pools, queue depth, storage capacity.
• Business indicators: completed orders, payment failures, or document-processing rate.

The four golden signals are latency, traffic, errors, and saturation. They provide a strong starting point but do not replace domain-specific measures. Percentiles are often more informative than averages because a small population of slow requests may be hidden by the mean. Percentiles must be calculated over a clearly defined population and interval; a p99 computed separately on many small instances cannot always be combined accurately by averaging the p99 values.

Metric labels require control. High-cardinality labels such as user ID or unrestricted URL can create excessive storage and query cost. Aggregation should preserve operational meaning without exposing sensitive information.

3.2 Logs

Logs are records of discrete events. High-quality structured logs include a timestamp with a stated time standard, severity, service identity, environment, request or trace identifier, event name, and safe diagnostic fields. Logs should support investigation rather than merely record implementation details.

Logging design must address privacy and security. Passwords, authentication tokens, payment data, and unnecessary personal information should not be logged. Retention should be based on operational, legal, and regulatory requirements. Log integrity and access control are essential when logs may support forensic investigation or compliance evidence.

3.3 Distributed traces

A trace represents the path of a request through distributed components. It consists of spans, each describing an operation and its duration. Trace context propagation allows a request received by an API gateway to be correlated with downstream service, database, and queue operations.

Tracing is valuable for locating latency and dependency failures, but sampling is often required for cost control. Head sampling decides early whether to retain a trace; tail sampling can retain traces based on later properties such as errors or high latency. Trace data may contain sensitive arguments and must therefore be governed like other telemetry.

3.4 Alert design

An alert should be actionable, owned, urgent enough to interrupt normal work, and linked to a runbook. Alerting on every low-level event produces fatigue and encourages ignoring alerts. Symptom-based alerts, such as a sustained SLO violation or rapidly increasing queue delay, often better reflect user impact than alerts on isolated causes.

A useful alert includes: what is wrong, affected scope, severity, start time, relevant dashboard, probable causes, immediate mitigations, and escalation path. Alert thresholds should be tested against normal variation and known failure scenarios.

4. Performance Engineering and Capacity Planning

Performance concerns the time and resources required to produce useful service behavior. Important dimensions include latency, throughput, concurrency, resource utilization, and scalability. A service can have high throughput but unacceptable latency, or low latency under light load but collapse under concurrency.

4.1 Basic capacity relationships

A fundamental relationship for a stable system is Little’s Law:

L = λW

where L is the average number of items in the system, λ is the average arrival rate, and W is the average time an item spends in the system. The relationship assumes a stable system over the measurement interval and consistent definitions of arrivals and completions. For a web service, if the average arrival rate is 200 requests per second and average response time is 0.25 seconds, the average number of in-flight requests is:

L = 200 × 0.25 = 50 requests.

This is not a direct formula for CPU capacity. It describes a queueing relationship and must be combined with workload tests and resource measurements.

A simple utilization measure for a resource is:

Utilization = busy time / available time.

As utilization approaches the practical limit of a bottleneck, queueing delay may rise sharply. Therefore, capacity planning should not simply target 100% CPU. It should establish safe operating ranges, account for failure of a component, and reserve headroom for bursts and deployments.

4.2 Capacity-planning method

A defensible planning process includes:

1. Define workload drivers, such as requests, active users, data volume, or scheduled jobs.

2. Measure baseline demand and identify seasonal or event-driven peaks.

3. Establish performance objectives and resource limits.

4. Load-test representative workloads, including dependency behavior and failure modes.

5. Model growth, uncertainty, and redundancy requirements.

6. Compare planned capacity with cost and operational constraints.

7. Review forecasts against actual demand and recalibrate.

Suppose a service currently receives 1,000 requests per second and is expected to grow by 20% per month for three months. The forecast is:

1,000 × 1.2³ = 1,728 requests per second.

If one instance safely handles 250 requests per second under the required latency objective, the nominal instance requirement is 1,728 / 250 = 6.912, so at least 7 instances are required before redundancy and headroom. If the design requires one instance to be unavailable while maintaining capacity, at least 8 instances are needed merely to tolerate that loss. Additional headroom may justify a larger fleet. The calculation assumes the per-instance benchmark remains valid at the forecast workload and that instances scale approximately linearly; real systems often violate both assumptions.

4.3 Autoscaling

Autoscaling changes capacity in response to demand or predicted demand. Horizontal scaling adds instances or containers; vertical scaling increases resources allocated to an instance; scheduled scaling anticipates known patterns; predictive scaling uses forecasts.

A common target-utilization rule is:

Desired capacity = current capacity × current measured load / target load.

For example, if 4 instances are running at 80% average CPU and the target is 50%, the estimated desired capacity is 4 × 0.80 / 0.50 = 6.4, which must be rounded according to the platform’s scaling policy, normally to at least 7 instances. This formula is a control heuristic, not a guarantee of performance. CPU may not be the limiting resource, measurements may lag, and abrupt changes can cause oscillation.

Good autoscaling includes stabilization windows, scale-out and scale-in limits, cooldown periods, health checks, and protection against runaway cost. Queue-based workers should often scale on queue age or backlog per worker rather than CPU alone. Databases may require connection, I/O, lock, or storage metrics, and may not support rapid horizontal scaling. Autoscaling cannot compensate for an inefficient algorithm, an overloaded dependency, or an architectural bottleneck.

5. Reliability Patterns and Change Operations

Reliability is strongly influenced by change. Many incidents result from software releases, configuration changes, infrastructure modifications, or dependency updates rather than from random hardware failure.

Blue-green deployment maintains two environments and directs traffic to one at a time. It enables rapid switching but may require double capacity and careful data-schema compatibility. Canary deployment sends a small proportion of traffic to a new version and compares its behavior with the baseline. It reduces blast radius but requires representative traffic, reliable metrics, and a rollback mechanism. Rolling deployment replaces instances gradually and is resource-efficient, but mixed-version operation can create compatibility problems.

Safe changes commonly use:

• automated tests and policy checks;
• versioned configuration and infrastructure as code;
• progressive exposure;
• backward-compatible database migrations;
• health checks based on meaningful service behavior;
• automatic or operator-approved rollback;
• change records linked to observed incidents.

A circuit breaker prevents repeated calls to a failing dependency by opening the circuit after defined failures, then probing cautiously during a half-open state. Timeouts prevent indefinite resource occupation. Retries should be bounded, use backoff and jitter, and be applied only when operations are safe to repeat. Uncontrolled retries can create a retry storm and worsen an outage. Bulkheads isolate resource pools so that failure in one workload does not exhaust resources for all workloads.

6. Incident Management and Learning

Incident management coordinates detection, response, communication, mitigation, recovery, and learning. Roles may include an incident commander, operations lead, communications lead, and subject-matter experts. During a high-severity incident, role clarity is more valuable than an unstructured group of technically capable people.

The first response should establish impact and scope, protect evidence, mitigate user harm, and communicate honestly. Mitigation is often preferable to immediate root-cause repair: disabling a nonessential feature, shifting traffic, rolling back a change, or serving cached content may restore service while investigation continues.

A blameless post-incident review focuses on system conditions and decisions rather than individual fault. It should document timeline, detection, customer impact, contributing factors, what worked, what failed, and prioritized corrective actions with owners and due dates. “Improve monitoring” is weak; “create an alert when checkout success falls below 99.5% for five minutes and test it in a controlled failure exercise” is actionable.

7. Chaos Engineering

Chaos engineering is the disciplined experimentation on a system to learn how it behaves under realistic adverse conditions. It is not random destruction. A safe experiment has a hypothesis, defined steady-state indicators, limited blast radius, authorization, abort conditions, observation plan, and review.

An example hypothesis is: “If one availability zone becomes unreachable, the order service will maintain at least 99.5% successful requests and p99 latency below 1 second for existing traffic.” The experiment may begin in a non-production environment, then move to a small production scope if controls are adequate. It should test not only infrastructure failure but also dependency latency, credential expiry, queue buildup, clock issues, and capacity exhaustion where appropriate.

Chaos experiments reveal hidden assumptions: a supposedly redundant service may share a network dependency; failover may work technically but violate data-loss objectives; autoscaling may be too slow; or alerts may detect failure without enabling recovery. Experiments must respect safety, privacy, contractual obligations, and change-control requirements.

8. Backup, Disaster Recovery, and Business Continuity

Backup is the creation of recoverable copies of data or configuration. Disaster recovery (DR) is the restoration of technology services after a disruptive event. Business continuity (BC) is broader: it ensures that critical business functions continue at an acceptable level, including people, facilities, suppliers, communications, manual workarounds, and technology.

Two central DR objectives are:

• Recovery point objective (RPO): the maximum acceptable amount of data loss measured in time.
• Recovery time objective (RTO): the maximum acceptable time to restore a service or business function.

If an application has an RPO of 15 minutes, the recovery design must limit potential data loss to approximately 15 minutes under the stated failure scenario. If its RTO is 2 hours, the service must be restored within 2 hours, including detection, decision-making, infrastructure provisioning, data recovery, validation, and traffic restoration. RPO and RTO are requirements, not descriptions of a backup product.

Backup strategies include full, incremental, and differential backups, continuous replication, snapshots, and logical exports. A snapshot may be fast but can be unsuitable as the only recovery mechanism if it is inaccessible during a regional compromise or if it lacks application-consistent state. Backups should be encrypted, access-controlled, retained according to policy, separated from production credentials, and periodically restored in tests. The principle of 3-2-1 backup—at least three copies, on at least two media or storage types, with at least one copy off-site—is a useful baseline, but cloud implementations must interpret “media” and independence carefully.

DR patterns include:

• Backup and restore: lowest standby cost, but usually longest recovery time.
• Pilot light: core components are prepared while capacity is scaled during recovery.
• Warm standby: a reduced-capacity environment runs continuously.
• Active-active: multiple sites serve traffic, providing fast failover but higher complexity and cost.

Recovery procedures must address dependencies, secrets, DNS or routing, identity, schemas, queues, certificates, observability, and human approvals. A plan that restores compute but not encryption keys or database consistency is not a complete DR plan.

Business-impact analysis ranks services according to the consequences of interruption. Dependencies should be mapped by business process, not only by technical architecture. Continuity plans should include alternate suppliers, communication channels, emergency authority, regulatory notifications, and manual procedures where feasible.

9. Governance, Security, and Responsible Operations

Reliability operations involve governance because outages can affect safety, finances, privacy, and public trust. Access to production systems should follow least privilege, separation of duties, strong authentication, and auditable approval. Emergency access should be time-limited and reviewed afterward.

Operational data may contain personal or confidential information. Telemetry collection should follow data minimization, retention limits, access control, and applicable privacy law. Regulatory requirements may impose recovery, retention, incident-notification, resilience, or audit obligations. Organizations should not claim a recovery capability that has not been tested.

Cost is also a reliability concern. Budget controls, quota policies, anomaly detection, and tagging help prevent an operational success from becoming a financial incident. AI-assisted operations can summarize incidents, identify anomalous patterns, recommend remediation, or generate runbooks. However, AI outputs can be incorrect, explanations can be incomplete, and automated remediation can amplify an error. Human approval, bounded actions, audit trails, evaluation datasets, and rollback are essential for high-impact operations.

10. Illustrative Case: Multi-Region Payment Platform

Background

Consider an illustrative cloud payment platform serving online retailers. It uses an API gateway, payment service, fraud service, relational database, message queue, and external banking providers. Its business requirement is high checkout availability, accurate payment state, and controlled recovery after a regional failure.

Problem

The platform reports 99.95% monthly API availability, but customers still experience failed checkouts. Investigation shows that the availability metric counts HTTP 200 responses even when an asynchronous payment message is later lost. The service also has high p99 latency during promotional events because autoscaling follows CPU while the actual bottleneck is the database connection pool.

Analysis

The team redesigns its SLIs. Checkout success is measured from accepted checkout requests that result in a durable, correctly correlated payment state, not merely an HTTP response. It adds latency and queue-age indicators and traces the request from the gateway through payment, fraud, database, and provider calls. A dependency map shows that all regions use a single externally hosted fraud endpoint.

Load testing demonstrates that one application instance safely processes 250 checkout requests per second under the target latency, but the database reaches its connection limit before application CPU becomes high. A regional failure exercise also reveals that database replication is asynchronous, creating a possible data-loss interval.

Approach and solution

The platform adopts an internal checkout SLO based on successful durable payment state, with a separate latency SLO. Error-budget policy pauses high-risk feature releases when the budget is consumed rapidly. Autoscaling uses request rate for the stateless tier and connection-pool saturation plus queue age for workers. Connection limits are explicitly budgeted across services.

The deployment process uses canaries with automatic rollback when checkout success, p99 latency, or payment reconciliation errors cross tested thresholds. Timeouts and bounded retries are applied to the banking provider; payment operations use idempotency keys so retries do not create duplicate charges. A circuit breaker and queue isolate provider failure from the user-facing API.

Backups are encrypted and copied to an independently controlled recovery account. Database restoration is tested monthly, and a regional recovery exercise is performed quarterly. The documented DR plan specifies an RTO of 60 minutes and an RPO of 5 minutes for the payment ledger, subject to provider and replication assumptions. Because the fraud service is a critical dependency, the business approves a degraded mode using cached risk rules and post-authorization review when the provider is unavailable.

Outcome

The revised indicators expose failures that the former availability metric hid. Promotional-event performance improves because scaling responds to the real bottleneck. Idempotency and reconciliation reduce duplicate-payment risk, while tested recovery procedures make the stated RTO and RPO credible rather than aspirational.

Learning points

1. A technically successful HTTP response is not necessarily a successful business transaction.

2. Scaling metrics must represent the resource that limits performance.

3. Dependencies, data semantics, and recovery validation are as important as infrastructure redundancy.

4. SLOs and error budgets connect engineering decisions with business risk.

5. Recovery objectives must be demonstrated through restoration and failover exercises.

Challenges and Limitations

Reliability measurement can encourage local optimization. Teams may improve an SLO while users suffer from an unmeasured failure mode. Observability can become expensive and can create privacy risk. Distributed tracing and logs are incomplete when instrumentation is missing or sampling excludes important events. Capacity forecasts are uncertain, especially for new products and nonlinear workloads.

Cloud provider abstractions reduce operational burden but can obscure shared dependencies, quota limits, regional constraints, and provider-level failure domains. Multi-region designs may improve continuity but introduce data consistency, compliance, networking, and operational complexity. Chaos engineering cannot prove that a system will survive every failure; it provides evidence about tested hypotheses. Finally, automation reduces toil only when the underlying controls, tests, ownership, and rollback mechanisms are mature.

Contemporary Best Practices

Effective cloud operations generally include:

• user-centered SLIs with explicit measurement definitions;
• separate indicators for availability, latency, correctness, freshness, and durability;
• dashboards for exploration and alerts for actionable conditions;
• structured, privacy-aware telemetry with correlation identifiers;
• capacity tests that include dependencies and failure conditions;
• autoscaling based on bottleneck-relevant signals;
• progressive delivery and tested rollback;
• bounded retries, timeouts, idempotency, circuit breakers, and bulkheads;
• immutable, encrypted, isolated backups with regular restore tests;
• documented RTO and RPO validated by exercises;
• blameless post-incident learning with tracked corrective actions;
• controlled chaos experiments;
• least privilege, auditability, cost governance, and clear ownership.

Chapter Summary

Reliable cloud operations combine prevention, detection, response, recovery, and learning. SRE supplies a practical framework through SLIs, SLOs, SLAs, error budgets, automation, and reduction of toil. Observability integrates metrics, logs, traces, and events so that operators can understand distributed system behavior. Performance engineering requires attention to latency, throughput, concurrency, saturation, and bottlenecks; Little’s Law provides a useful queueing relationship when its stability assumptions hold.

Capacity planning must combine forecasts, load tests, redundancy, uncertainty, and cost. Autoscaling is a control mechanism rather than a substitute for sound architecture. Progressive delivery, timeouts, bounded retries, circuit breakers, bulkheads, and idempotency reduce the blast radius of change and dependency failure. Chaos engineering tests resilience hypotheses under controlled conditions.

Backup, disaster recovery, and business continuity address different but related needs. RPO describes acceptable data loss, while RTO describes acceptable restoration time. Both must be validated through realistic exercises. Governance, privacy, security, cost management, and responsible use of AI are integral to cloud operations. The strongest operational programs connect technical measures to user outcomes and business impact.

Key Terms

SRE: Site reliability engineering; an engineering discipline for reliable operation of services.

Reliability: The ability of a system to perform its required function under stated conditions for a specified period.

Availability: The proportion of defined time or eligible requests during which a service is usable.

Resilience: The ability to withstand disruption and recover while maintaining acceptable service.

Observability: The ability to infer internal system state from external outputs.

SLI: Service-level indicator; a quantitative measure of user-relevant service behavior.

SLO: Service-level objective; a target for an SLI over a defined period.

SLA: Service-level agreement; a contractual or business commitment concerning service performance.

Error budget: The permitted amount of unreliability implied by an SLO.

Toil: Repetitive, manual, automatable operational work that provides little enduring value.

Telemetry: Operational data such as metrics, logs, traces, and events.

Latency: The time taken to complete an operation or produce a response.

Saturation: The degree to which a resource is approaching its effective capacity.

Autoscaling: Automatic adjustment of service capacity in response to demand or forecasts.

Chaos engineering: Controlled experimentation to learn system behavior under adverse conditions.

RPO: Recovery point objective; the maximum acceptable data-loss interval.

RTO: Recovery time objective; the maximum acceptable restoration interval.

Disaster recovery: Restoration of technology services after a disruptive event.

Business continuity: Maintenance of critical business functions during and after disruption.

Idempotency: A property whereby repeating an operation produces the same intended result as performing it once.

Review Questions

Short Answer Questions

1. Distinguish reliability, availability, resilience, and recoverability.

2. What are an SLI, an SLO, and an SLA?

3. Why can percentile latency be more useful than average latency?

4. Define RPO and RTO.

5. What is the purpose of a circuit breaker?

Long Answer Questions

1. Explain how error budgets influence release and reliability decisions.

2. Describe an observability strategy for a distributed cloud application using metrics, logs, and traces.

3. Explain the major steps in capacity planning and identify assumptions that may make forecasts inaccurate.

4. Compare backup-and-restore, warm-standby, and active-active disaster-recovery strategies.

5. Discuss how progressive delivery, timeouts, retries, idempotency, and bulkheads work together to reduce operational risk.

Analytical / Application Questions

1. A service receives 400 requests per second and has an average response time of 0.15 seconds. Use Little’s Law to estimate the average number of in-flight requests. State the assumption required for the calculation.

2. A system has an SLO of 99.95% over a 30-day month. Calculate the error-budget percentage and explain why this aggregate target may not detect a short, severe outage.

3. Design a controlled chaos experiment for a cloud service that depends on a database and an external API. Specify the hypothesis, steady-state indicators, blast-radius controls, abort conditions, and learning outcome.

Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes an SLO from an SLA?

A. An SLO is an internal or operational target, while an SLA is generally a business or contractual commitment.

B. An SLO records logs, while an SLA records traces.

C. An SLO applies only to databases, while an SLA applies only to networks.

D. An SLO is a backup copy, while an SLA is a recovery procedure.

Correct Answer: A. An SLO is an internal or operational target, while an SLA is generally a business or contractual commitment.

Question 2

A service has an availability SLO of 99.9%. What is its error-budget fraction?

A. 0.001%

B. 0.01%

C. 0.1%

D. 1%

Correct Answer: C. 0.1%

Question 3

Which telemetry type is primarily designed to follow one request across multiple distributed services?

A. Structured log

B. Distributed trace

C. Capacity forecast

D. Backup snapshot

Correct Answer: B. Distributed trace

Question 4

A service receives 200 requests per second and has an average response time of 0.25 seconds. Under Little’s Law, what is the average number of in-flight requests?

A. 25

B. 50

C. 200

D. 800

Correct Answer: B. 50

Question 5

If four instances operate at 80% utilization and the target utilization is 50%, what estimated capacity results from the proportional autoscaling heuristic?

A. 5.0 instances

B. 6.4 instances

C. 7.0 instances after rounding up

D. 10.0 instances

Correct Answer: C. 7.0 instances after rounding up

Question 6

Which practice most directly limits damage from repeated retries against a failing dependency?

A. Unbounded immediate retries

B. Bounded retries with backoff and jitter

C. Removing all request timeouts

D. Increasing log verbosity only

Correct Answer: B. Bounded retries with backoff and jitter

Question 7

What does an RPO primarily specify?

A. Maximum acceptable data-loss interval

B. Maximum CPU utilization

C. Maximum number of service instances

D. Minimum encryption key length

Correct Answer: A. Maximum acceptable data-loss interval

Question 8

Which characteristic is essential for a valid chaos-engineering experiment?

A. An undefined blast radius

B. A hypothesis, steady-state indicators, and abort conditions

C. A guarantee that no system component is affected

D. A requirement to run only during an outage

Correct Answer: B. A hypothesis, steady-state indicators, and abort conditions

Question 9

Why is an HTTP 200 response not necessarily a sufficient checkout SLI?

A. HTTP 200 responses cannot be logged.

B. A response may be returned even though the payment state is not durably and correctly recorded.

C. HTTP status codes measure CPU utilization.

D. HTTP 200 always indicates a database failure.

Correct Answer: B. A response may be returned even though the payment state is not durably and correctly recorded.

Question 10

Which disaster-recovery pattern generally has the highest continuously running cost and can provide rapid failover?

A. Backup and restore

B. Pilot light

C. Warm standby

D. Active-active

Correct Answer: D. Active-active