3. Service-Level Indicators, Objectives and Error Budgets

Introduction and Context

Site Reliability Engineering (SRE) treats reliability as an engineering property that must be measured, designed, and managed rather than assumed. A service may be described as “highly available” or “fast,” but such descriptions are too vague to guide engineering decisions. Reliability becomes operationally useful only when user experience is translated into observable measurements, explicit targets, and decision rules.

Service-level indicators (SLIs), service-level objectives (SLOs), service-level agreements (SLAs), and error budgets provide this measurement and decision framework. Together, they connect technical operations with product priorities, customer expectations, risk management, and organizational governance. They help teams answer questions such as: What does reliability mean for this service? How much failure is acceptable? When should a release be delayed? When is additional capacity or architectural investment justified?

The central idea is not that every service must be perfect. Absolute reliability is usually impossible or economically irrational. Instead, an organization should establish a reliability level that is valuable to users and sustainable for the business. An error budget makes the permitted unreliability explicit, allowing teams to balance innovation against operational risk.

Learning Objectives

By the end of this chapter, learners should be able to:

1. Distinguish SLIs, SLOs, SLAs, and error budgets.

2. Design user-centred indicators for availability, latency, throughput, correctness, durability, and freshness.

3. Select appropriate measurement events, populations, aggregation methods, and time windows.

4. Calculate compliance and error budgets for common service objectives.

5. Design multi-window, multi-burn-rate alerts.

6. Develop error-budget policies that guide releases, incident response, and investment.

7. Evaluate the technical, commercial, ethical, and governance implications of reliability targets.

8. Identify common measurement errors and improve the quality of reliability decisions.

From User Experience to Measurable Reliability

Reliability is not identical to uptime. A service can be reachable while returning incorrect data, responding too slowly, or serving stale information. Conversely, a brief infrastructure failure may have little user impact if requests are rapidly retried or transparently served from a replica. SRE therefore begins with the user-visible outcome rather than with the internal component being monitored.

An SLI is a quantitative measure of a service aspect that matters to users. It is normally expressed as the proportion of successful events among all relevant events:

SLI = good events / valid events

For example, if 997,000 of 1,000,000 valid requests return a correct response within the defined latency threshold, the request-success SLI is 99.7 percent. The definition of “good,” “valid,” and “event” is part of the SLI design and must be explicit.

An SLO is a target value or range for an SLI over a specified period. “99.9 percent of valid checkout requests should complete successfully during each calendar month” is an SLO. The time window matters because the same percentage has different operational implications over a day, month, or year.

An SLA is a formal commitment made to customers or other parties. It may include service targets, exclusions, reporting rules, remedies, and financial credits. An SLA is therefore a business and contractual instrument, whereas an SLO is primarily an internal reliability target. An organization commonly sets its internal SLO stricter than its external SLA to preserve a margin for investigation and remediation.

The error budget is the amount of unreliability permitted by an SLO. If the objective is 99.9 percent good events, the error budget is 0.1 percent of valid events. For a monthly time-based availability objective, the budget can also be expressed as allowable unavailability. The error budget is not permission to cause failures deliberately; it is a risk allowance used to make trade-offs explicit.

Designing Meaningful SLIs

Availability and request success

Availability can be measured as the proportion of valid requests that produce an acceptable response. A strong request-based SLI usually considers more than the absence of a server error. A response may have status code 200 and still contain an invalid price, incomplete result, or unauthorized data. The definition should therefore reflect the user’s successful task where feasible.

A basic availability SLI is:

Availability = successful requests / valid requests

The measurement must specify whether probes, retries, duplicate requests, health checks, and traffic from internal clients are included. Synthetic checks are useful for detecting a total outage, but real-user requests usually provide a more representative view of experienced reliability.

Latency

Latency is the time between a defined request event and a defined completion event. The definition should identify whether it includes network transfer, queueing, server processing, downstream calls, and client rendering. Mean latency is often misleading because a small number of very slow requests may seriously affect users while having little effect on the average.

Percentiles are generally more informative. A latency objective might require 99 percent of search requests to complete within 500 milliseconds. The percentile must be calculated over a meaningful population and window. Aggregating averages from multiple hosts can conceal tail latency, and averaging already aggregated percentiles is mathematically invalid in most cases.

Latency objectives should be linked to user tasks. A background batch job and an interactive payment request should not necessarily have the same latency target. When a request is intentionally asynchronous, the SLI may measure acknowledgement time separately from completion time.

Throughput and capacity

Throughput measures the volume of work completed per unit of time, such as messages processed per second, orders completed per hour, or database writes per minute. Throughput alone is not a reliability indicator: a system may process a high volume incorrectly. It becomes useful when paired with correctness, latency, and loss measurements.

For capacity planning, teams may track accepted requests, completed requests, queue growth, and saturation. A throughput SLI should distinguish demand from successful service. Measuring only requests received can reward a system for accepting work that it never completes.

Correctness

Correctness measures whether the system produces the right result. It may be evaluated through invariants, reconciliation, validation rules, sampled comparisons, or domain-specific outcomes. Examples include the percentage of financial transactions whose ledger entries reconcile, inventory calculations that match authoritative records, or recommendations that satisfy policy constraints.

Correctness is often harder to measure than availability because the expected answer may not be immediately known. Teams may use delayed verification, reference datasets, dual computation, checksums, or user-confirmed outcomes. A system that is available but systematically wrong can create greater harm than one that fails visibly.

Durability and data loss

Durability concerns whether accepted data remains available and uncorrupted for the required period. A durability SLI might measure the proportion of successfully acknowledged objects that remain retrievable and intact after a specified interval. It may also track unrecoverable records, replication failures, restore success, and recovery-point gaps.

Durability objectives must define the unit of measurement and the observation period. “No data loss” is an aspiration, not a measurable specification, unless the organization defines what counts as accepted data, what recovery evidence is required, and what time horizon applies.

Freshness

Freshness measures whether information is no older than the user or business process permits. It is especially important for dashboards, search indexes, fraud signals, inventory, and replicated databases. A freshness SLI can be expressed as the proportion of observations whose age is below a threshold, or as the age distribution of the newest successfully processed event.

Freshness should not be confused with availability. A dashboard may load successfully while displaying data that is several hours old. The appropriate objective may combine availability, latency, and freshness—for example, “99 percent of user-visible inventory views load within two seconds and reflect data no more than five minutes old.”

Measurement Boundaries and Aggregation Choices

A credible SLI requires a clearly defined measurement contract. This contract should identify the event population, exclusions, source of truth, sampling method, aggregation, and treatment of retries. Ambiguous definitions produce arguments rather than useful decisions.

The event population should represent real user demand. Excluding failed requests because they came from a busy region, a particular client version, or a period of high load can create an artificially positive result. Exclusions may be legitimate—for example, deliberately malformed requests—but they must be documented and consistently applied.

Measurement can occur at the client, edge, service, or business-transaction layer. Client-side measurements best capture user experience but may be affected by device and network conditions. Server-side measurements offer greater control and diagnostic detail but may miss failures before requests reach the service. Mature programs use multiple views while selecting one authoritative SLI for policy decisions.

Aggregation also matters. A service-level availability figure can be calculated by event count or by time. Request-based measurement gives high-volume traffic greater influence, while time-based measurement treats each interval equally. Neither is universally correct. A low-volume but business-critical operation may require a separate SLI so that its importance is not hidden by a large volume of routine traffic.

Objectives, Windows, and Error-Budget Mathematics

An SLO must specify four elements: the measured property, the good-event definition, the target, and the evaluation window. For example: “At least 99.95 percent of authenticated API requests shall return a correct response within 800 milliseconds over a rolling 28-day window.”

A rolling window updates continuously and is useful for operational control. A calendar window is easier to report and align with billing or governance processes, but it can create boundary effects. A service that consumes its entire budget on the last day of a month may appear compliant again on the first day without having improved. Some organizations therefore use both a reporting window and a rolling operational window.

If an SLO target is 99.9 percent, the error-budget fraction is 0.1 percent. For 2,000,000 valid requests, the budget is 2,000 failed or otherwise bad events. If 600 bad events have occurred, 30 percent of the event budget has been consumed. For a time-based 99.95 percent objective over a 30-day month, the nominal budget is 0.05 percent of 43,200 minutes, or 21.6 minutes.

Event-based and time-based budgets should not be casually substituted. A five-minute outage during a low-traffic period may consume nearly all of a time-based budget but relatively few request-based errors. The selected model should correspond to the user and business impact.

SLOs also need a scope. A single global objective may hide regional, tenant, endpoint, or customer-specific failures. Conversely, too many objectives create administrative complexity and encourage teams to optimize whichever metric is easiest. The objective set should be small enough to govern and rich enough to represent meaningful user journeys.

Burn Rate and Alert Design

Error-budget consumption is often more actionable than a raw error percentage. Burn rate describes how quickly the budget is being spent relative to the rate that would exhaust it exactly at the end of the SLO window.

A burn rate of 1 means the service is consuming budget at the sustainable average rate. A burn rate of 10 means it is consuming budget ten times faster and would exhaust the entire budget in approximately one-tenth of the window, assuming the rate continued.

Alerting directly on a single threshold frequently creates either noise or delayed detection. A better approach uses multi-window, multi-burn-rate alerts. A long window confirms that a substantial fraction of the budget has been consumed, while a short window detects a recent severe event. An alert may require both conditions, reducing false positives caused by a brief spike or by a slow, low-level degradation.

The exact windows and thresholds depend on the SLO period, service criticality, and response capability. A common design principle is to pair a short window with a high burn-rate threshold for rapid incident response and a longer window with a lower threshold for emerging problems. Alerts should identify the affected SLO, current burn rate, budget remaining, likely scope, and recommended action. They should not merely report that a generic error percentage is high.

Alert thresholds must account for traffic volume. A low-volume service may show a dramatic percentage change from a single failed request. Statistical techniques, minimum event counts, and severity classification can prevent disproportionate responses. Alert fatigue is itself a reliability risk because responders begin ignoring meaningful signals.

Error-Budget Policies as Operating Agreements

An error-budget policy translates measurement into organizational action. It should state what happens when budget consumption reaches defined levels. For example, normal release activity may continue while sufficient budget remains; a declining budget may require additional testing, staged rollout, or senior review; exhausted budget may pause risky changes except for reliability fixes and urgent security work.

The policy should not be a blunt prohibition on all change. Freezing every deployment after a minor event can create operational stagnation, delay security remediation, and encourage teams to manipulate measurements. A mature policy distinguishes between changes that increase risk and changes that reduce it. It may also permit low-risk documentation, rollback, capacity, compliance, or security work during a freeze.

Error budgets support negotiation between product and engineering. Product teams obtain a clear allowance for experimentation, while engineering teams obtain a defensible basis for requesting reliability investment. The policy should specify decision rights, escalation paths, exceptions, and the evidence required to restore normal release velocity.

Budget policies are most effective when reviewed at service boundaries. A platform team may own infrastructure SLOs, while a product team owns user-journey SLOs. Shared dependencies require explicit agreements about how failures are attributed and how one team’s budget affects another’s.

SLAs, Governance, and Commercial Commitments

An SLA should be derived from actual user and business needs, not copied mechanically from an internal dashboard. It should define the service covered, measurement method, service credits or remedies, maintenance exclusions, customer responsibilities, reporting period, and dispute process. Ambiguous SLA language creates legal and operational conflict.

The relationship among the three targets can be represented as follows: an internal SLO is the engineering objective; an error budget is the operational allowance derived from that objective; and an SLA is the externally enforceable commitment. The SLO may be stricter than the SLA because the organization needs room to detect, investigate, and correct degradation before a contractual breach occurs.

Governance also involves fairness and transparency. Reliability reports should not hide failures affecting small customers, particular regions, accessibility technologies, or less common workflows. If a measurement excludes a class of users, the exclusion should be justified and disclosed to relevant stakeholders. Reliability metrics can influence compensation, performance evaluation, and customer trust; therefore, their design must resist gaming.

Applied Case: Reliability Governance for an Online Retail Checkout

Background

The following is an illustrative case. A growing online retailer operates a checkout service connected to payment authorization, inventory reservation, taxation, and order creation systems. The company historically reported “uptime” from a load balancer health check. Although the dashboard showed 99.99 percent uptime, customers regularly reported abandoned carts and duplicate payment attempts.

Situation and problem

The health check confirmed only that the checkout process responded to a simple probe. It did not measure whether customers could complete a purchase, whether prices and inventory were correct, or whether payment authorization was reflected accurately in the order system. The organization also released major changes immediately before promotional events, when traffic and business risk were highest.

Analysis

The reliability team mapped the critical user journey and identified several distinct outcomes. A checkout request could fail explicitly, exceed a latency threshold, return an incorrect total, reserve inventory without creating an order, or create an order without durable payment reconciliation. No single metric could represent all of these risks.

The team designed a set of SLIs: successful completion of valid checkout attempts; percentage of checkout responses completed within four seconds; correctness of calculated totals compared with authoritative pricing rules; successful reconciliation of payment and order records; and freshness of inventory data. A separate availability SLI was retained for the supporting API, but the user-journey SLI became the principal release and governance measure.

Approach and solution

The organization established an internal monthly objective of 99.95 percent successful checkout attempts, stricter than the external contractual commitment for enterprise partners. It defined correctness failures as bad events even when the HTTP response was successful. For the payment-reconciliation objective, the team used a delayed verification process because some payment confirmations arrived asynchronously.

A rolling 28-day window was used for operational decisions, while monthly reports were retained for commercial review. The error-budget policy allowed normal releases when budget consumption was within the planned trajectory. If a high burn-rate alert occurred, risky deployments were paused, the latest change was evaluated for rollback, and the incident commander received a report showing affected regions and transaction stages. Exhausted budget triggered a review of release risk, capacity, dependency failure, and corrective work rather than an automatic punishment of the team.

Canary deployment was introduced for checkout changes. The canary was evaluated using the same user-centred SLIs, not merely infrastructure health. Automated checks also compared order totals and payment reconciliation results. A deployment could be halted when a short-window burn-rate alert indicated severe degradation, even if the monthly aggregate still appeared healthy.

Outcome

The revised measurement system exposed failures previously hidden by the health check. Teams could distinguish payment-provider problems from pricing defects and inventory staleness. Release decisions became more evidence-based, and product managers could evaluate the reliability cost of high-risk experiments. The organization did not eliminate all incidents; rather, it improved its ability to detect meaningful failures, prioritize remediation, and make explicit trade-offs.

Learning points

The case illustrates that availability is not sufficient for transaction-oriented services. User journeys often require several coordinated SLIs, and correctness or reconciliation may need delayed measurement. It also demonstrates why SLOs must be connected to deployment controls and why an error budget is valuable only when it changes decisions.

Challenges, Limitations, and Common Mistakes

A frequent mistake is selecting metrics because they are easy to collect rather than because they represent user value. CPU utilization, memory use, and container restarts are useful diagnostic signals but are usually poor primary SLIs. Another mistake is setting targets at 100 percent. A perfect target leaves no room for normal variation, measurement uncertainty, or safe change and can encourage unrealistic blame.

Teams may also optimize one dimension while damaging another. Improving latency by returning incomplete results is not reliability improvement. Increasing throughput by accepting work that later fails creates a misleading success story. Composite user journeys should therefore preserve the relationship between speed, availability, correctness, and durability.

Poorly designed exclusions, inappropriate sampling, percentile errors, missing low-volume traffic, and inconsistent treatment of retries can all distort results. Observability pipelines themselves may drop telemetry during an incident, creating the appearance of improvement. Measurement systems require testing, version control, ownership, and periodic audit.

Another limitation is that SLOs cannot capture every form of harm. Security breaches, privacy violations, regulatory failures, and catastrophic data corruption may be unacceptable even if they consume little statistical error budget. Error budgets must therefore complement, not replace, security controls, risk assessments, compliance obligations, and professional judgment.

Technology, Automation, and Emerging Practice

Modern observability platforms can calculate SLIs from logs, traces, metrics, event streams, and client telemetry. Distributed tracing helps connect a user transaction across multiple services, while high-cardinality dimensions allow analysis by region, endpoint, tenant, or software version. Care is required because excessive cardinality increases storage cost and can create privacy risks.

Reliability checks can be integrated into continuous delivery pipelines. A pipeline may compare canary performance with a baseline, calculate burn rate, verify correctness invariants, and automatically halt or roll back a deployment. Automated remediation should be bounded by safe controls; an automated rollback based on an unstable metric can amplify an incident.

Machine learning and anomaly detection can identify unusual latency, traffic, or freshness patterns, but anomaly detection is not a substitute for an SLO. An unusual condition may be harmless, while a predictable but unacceptable failure may not appear anomalous. AI-assisted operations should provide evidence, preserve audit trails, and avoid making irreversible changes without appropriate authorization.

Emerging practice increasingly emphasizes journey-level and outcome-based SLOs, continuous verification of deployment risk, and reliability views that incorporate dependency graphs. Organizations are also using reliability reviews to connect technical objectives with sustainability, accessibility, privacy, and resilience requirements.

Best Practices for Professional Use

1. Start with critical user journeys and business outcomes, not available infrastructure metrics.

2. Define good events, valid events, exclusions, retries, and aggregation rules in writing.

3. Use a small, meaningful set of SLIs covering relevant dimensions such as success, latency, correctness, durability, and freshness.

4. Set internal SLOs separately from external SLAs and preserve operational margin.

5. Use rolling windows for operational control where boundary effects would be harmful.

6. Combine short and long burn-rate windows to balance rapid detection with alert quality.

7. Link error-budget consumption to release, incident, investment, and escalation decisions.

8. Review SLOs when architecture, user expectations, traffic patterns, or contractual commitments change.

9. Protect measurement systems from gaming, privacy violations, and unreviewed exclusions.

10. Treat reliability as a shared product, engineering, and governance responsibility.

Chapter Summary

SLIs translate user-visible service behaviour into measurable quantities. SLOs set explicit targets for those indicators over defined windows, while SLAs formalize selected commitments to customers. Error budgets express the permitted unreliability and provide a practical mechanism for balancing innovation with operational risk.

Effective SLI design covers the outcomes that matter: availability, latency, throughput, correctness, durability, and freshness. Measurement boundaries, event populations, aggregation methods, retries, exclusions, and windows must be defined carefully. Burn-rate alerts improve detection by combining the severity and persistence of budget consumption. Error-budget policies make reliability operational by guiding deployments, incident response, escalation, and investment.

The most important lesson is that reliability is not represented adequately by uptime alone. A professional SRE program measures complete user journeys, distinguishes internal objectives from contractual promises, and uses evidence to manage trade-offs. Metrics must be governed responsibly because they influence customer trust, organizational behaviour, and technical priorities.

Key Terms

Service-Level Indicator (SLI): A quantitative measurement of a user-relevant aspect of service performance.

Service-Level Objective (SLO): A target value or range established for an SLI over a specified period.

Service-Level Agreement (SLA): A formal external commitment defining service targets, remedies, and measurement conditions.

Error budget: The permitted amount of unreliability implied by an SLO.

Good event: An event that satisfies the defined success conditions of an SLI.

Valid event: An event included in the denominator of an SLI calculation.

Availability: The proportion of valid requests or service time that remains usable.

Latency: The elapsed time between defined request and completion events.

Percentile: A statistical value below which a specified proportion of observations fall.

Throughput: The amount of work successfully processed per unit of time.

Correctness: The degree to which a service produces the expected or valid result.

Durability: The ability to preserve accepted data accurately for the required period.

Freshness: The degree to which information remains within an acceptable age threshold.

Rolling window: A continuously advancing period used to evaluate an objective.

Calendar window: A fixed period, such as a month, used for measurement or reporting.

Burn rate: The rate at which error budget is being consumed relative to its sustainable rate.

Multi-window alert: An alert requiring evidence from more than one time window.

User journey: A sequence of service interactions that accomplishes a user goal.

Canary deployment: A limited release used to evaluate a change before wider deployment.

Review Questions

Short Answer Questions

1. What is the difference between an SLI and an SLO?

2. Why is uptime alone an incomplete measure of reliability?

3. Define an error budget and explain how it is derived from an SLO.

4. What is meant by burn rate?

5. Why might a service need a freshness SLI in addition to an availability SLI?

Long Answer Questions

1. Explain how availability, latency, correctness, durability, and freshness can be measured for different service types.

2. Discuss the relationship among SLOs, SLAs, and error-budget policies.

3. Explain the design considerations involved in choosing an SLO measurement window.

4. Describe how multi-window, multi-burn-rate alerting improves operational response.

5. Analyze the organizational and managerial value of error budgets in balancing innovation and reliability.

Analytical / Application Questions

1. A service has a 99.9 percent monthly request-based SLO and receives 5,000,000 valid requests. Calculate its monthly error budget. If 7,500 bad events occur, assess the budget position and recommend an action.

2. Design a set of SLIs and SLOs for a digital banking money-transfer journey. Include availability, latency, correctness, durability, and freshness where appropriate.

3. A dashboard loads within two seconds for 99.99 percent of requests, but its data is often two hours old. Evaluate why the existing SLI is inadequate and propose a revised measurement model.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes an SLO from an SLA?

A. An SLO is an internal target, while an SLA is a formal external commitment.

B. An SLO measures infrastructure capacity, while an SLA measures source-code quality.

C. An SLO is always legally enforceable, while an SLA is never contractual.

D. An SLO defines an error event, while an SLA defines a software release.

Correct Answer: A. An SLO is an internal target, while an SLA is a formal external commitment.

Question 2

A service has a 99.9% request-based SLO and receives 2,000,000 valid requests in the evaluation window. What is its nominal error budget?

A. 20 bad events

B. 200 bad events

C. 2,000 bad events

D. 20,000 bad events

Correct Answer: C. 2,000 bad events

Question 3

Which measurement best represents the freshness of a replicated inventory view?

A. Percentage of requests returning HTTP 200

B. Percentage of observations whose data age is below five minutes

C. Average CPU utilization of the inventory service

D. Number of containers running during the day

Correct Answer: B. Percentage of observations whose data age is below five minutes

Question 4

Why are latency percentiles generally more useful than mean latency for an interactive service?

A. They eliminate the need to define a completion event.

B. They reveal the experience of slower portions of the request population.

C. They always require less telemetry storage.

D. They measure correctness automatically.

Correct Answer: B. They reveal the experience of slower portions of the request population.

Question 5

A burn rate of 10 indicates that, if the current failure rate continues, the service will consume its error budget at approximately what rate?

A. One-tenth of the sustainable rate

B. The sustainable rate exactly

C. Ten times the sustainable rate

D. One hundred times the sustainable rate

Correct Answer: C. Ten times the sustainable rate

Question 6

Which is the strongest reason for using a user-journey SLI for an online checkout?

A. It measures only the health of the load balancer.

B. It captures whether the customer can complete the intended business task.

C. It removes the need for dependency monitoring.

D. It guarantees that no transaction will ever fail.

Correct Answer: B. It captures whether the customer can complete the intended business task.

Question 7

What is a principal advantage of combining short and long windows in burn-rate alerting?

A. It removes the need for an SLO.

B. It detects severe recent events while confirming sustained budget impact.

C. It ensures all alerts are based on calendar months.

D. It converts an SLA into an SLI.

Correct Answer: B. It detects severe recent events while confirming sustained budget impact.

Question 8

Which practice is most likely to produce a misleadingly positive availability SLI?

A. Documenting valid-event exclusions

B. Including real-user requests in the measurement population

C. Excluding failed requests from a region because the region was under heavy load

D. Separating malformed requests from valid requests

Correct Answer: C. Excluding failed requests from a region because the region was under heavy load

Question 9

Why should an internal SLO often be stricter than an external SLA?

A. To provide operational margin before a contractual commitment is breached

B. To ensure that all services have identical objectives

C. To make the error budget equal to zero

D. To avoid measuring user-visible outcomes

Correct Answer: A. To provide operational margin before a contractual commitment is breached

Question 10

A dashboard responds quickly but displays data that is several hours old. Which conclusion is most accurate?

A. The dashboard is fully reliable because latency is low.

B. Availability and latency are adequate, but freshness is a separate reliability concern.

C. Freshness can be inferred from CPU utilization.

D. The dashboard must have a throughput failure.

Correct Answer: B. Availability and latency are adequate, but freshness is a separate reliability concern.