Operations, Observability and Reliability

Introduction

DevOps does not end when software is deployed. A deployed service must remain available, responsive, secure, recoverable and understandable while it operates in production. Operations is the discipline of running and supporting systems; observability is the ability to infer a system’s internal condition from its external outputs; reliability is the extent to which the system performs its intended function consistently under stated conditions.

Modern applications are distributed across virtual machines, containers, databases, networks, cloud services and third-party APIs. Failures may therefore arise from code defects, resource exhaustion, configuration changes, dependency failures, attacks or unexpected demand. Effective operations combines technical instrumentation with human processes: monitoring, alerting, incident response, capacity planning, service-level management and continual improvement.

This chapter presents a practical foundation for understanding logs, metrics, traces, dashboards and alerts. It also introduces service-level indicators, objectives and agreements; availability and resilience; incident management; post-incident learning; and disaster recovery.

Learning Objectives

By the end of this chapter, you should be able to:

1. Distinguish logs, metrics, traces and events, and select suitable uses for each.
2. Design useful dashboards and actionable alerts.
3. Define service-level indicators (SLIs), objectives (SLOs) and agreements (SLAs).
4. Calculate and interpret availability, error budgets and basic capacity measures.
5. Explain resilience, redundancy, graceful degradation and recovery strategies.
6. Participate effectively in incident response and blameless post-incident reviews.
7. Compare backup, disaster-recovery and business-continuity approaches.
8. Evaluate operational, ethical and governance risks in observability systems.

1. Operations as a DevOps Capability

Operations includes provisioning infrastructure, deploying releases, managing configurations, controlling access, monitoring health, responding to incidents, maintaining security, backing up data and retiring services. In a DevOps model, development and operations share responsibility for the service throughout its lifecycle. This is often described as “you build it, you run it,” although responsibility must be supported by training, automation and sustainable on-call arrangements.

A useful operational model has four connected activities:

• Prepare: define service requirements, dependencies, risks and recovery targets.
• Operate: run the service, observe it and manage routine changes.
• Respond: detect, assess, contain and recover from incidents.
• Learn: identify systemic improvements and update designs, tests and procedures.

Operational quality is not simply the absence of outages. A service can be available but slow, technically healthy but unusable, or operationally successful but unsafe because it exposes personal data in logs. Reliability must therefore be defined from the user’s perspective.

2. Observability: Understanding Systems Through Evidence

Observability is broader than monitoring. Monitoring asks whether known conditions are occurring—for example, whether CPU usage exceeds 90 percent. Observability asks whether available evidence is sufficient to explain unknown or unexpected conditions—for example, why only customers in one region experience failed checkouts.

2.1 Logs

A log is a time-stamped record of an event or state transition. Examples include an authentication failure, a payment request, a deployment, or a database connection error. Structured logs, commonly represented as JSON, are easier to search and aggregate than unstructured text.

A useful log record may contain:

• timestamp in a consistent timezone;
• severity level such as DEBUG, INFO, WARN or ERROR;
• service and environment;
• operation or event name;
• request or correlation identifier;
• outcome and error classification;
• relevant duration and status code;
• carefully controlled contextual fields.

Logs should support diagnosis rather than merely record activity. Repeatedly logging the same failure at several layers creates noise. Sensitive information such as passwords, access tokens, payment-card data and unnecessary personal identifiers must not be recorded.

2.2 Metrics

A metric is a numerical measurement observed over time. Common classes include:

The four golden signals are latency, traffic, errors and saturation. Infrastructure teams also use resource metrics such as CPU, memory, disk and network utilization. Metrics are efficient for alerting because they can be aggregated, but excessive label combinations create high cardinality and increase storage and query costs.

2.3 Traces

A distributed trace follows one transaction across services. A trace contains spans, where each span represents an operation such as an API call, cache lookup or database query. Trace context—usually a trace identifier and span identifier—allows related work to be connected across process boundaries.

Traces are especially useful when a request is slow or fails only through a particular dependency path. Sampling is often necessary because tracing every request may be expensive. Head-based sampling decides early; tail-based sampling can retain unusual or slow traces after observing their outcome.

2.4 Events and Profiles

An event records a significant occurrence, such as a deployment, scaling action or feature-flag change. Correlating events with metrics often reveals causation or timing relationships. Profiling examines how a program uses CPU, memory or locks and can identify performance problems that ordinary telemetry cannot explain.

2.5 The Telemetry Pipeline

Applications and infrastructure emit telemetry to agents or collectors. Collectors may enrich, filter, sample and route data to storage and analysis systems. A typical path is:

Application or platform → collector → processing and routing → metrics, log or trace backend → dashboard and alerting tools.

This pipeline is itself a production dependency. If telemetry collection fails, the service may continue operating while the organization becomes unable to understand it. Telemetry pipelines therefore require capacity limits, access control, retention policies and failure handling.

3. Dashboards and Alerts

A dashboard is a visual arrangement of operational information designed for a particular decision. A service dashboard should usually begin with user-facing indicators: request success rate, latency, traffic and availability. It can then show dependency health, saturation, recent deployments and current incidents. A dashboard containing hundreds of unrelated graphs is not necessarily informative.

Dashboards should answer three questions:

1. Is the service meeting its intended user outcome?
2. If not, which component or dependency is contributing to the problem?
3. What action should the responder take next?

An alert is a notification that requires investigation or action. Alerts should be based on symptoms that matter to users or on conditions requiring immediate intervention. An alert that fires frequently without meaningful action produces alert fatigue, leading responders to ignore or disable it.

Good alerts have a clear title, service, impact, start time, evidence, severity, runbook link and owner. They should include enough context to distinguish a transient event from a sustained incident. Warning thresholds may support investigation, while paging thresholds should normally indicate urgent impact.

A practical alert-quality test is: “If this alert fired at 03:00, would the responder know what decision or action is required?” If the answer is no, improve the alert, convert it to a dashboard signal or remove it.

4. SLIs, SLOs and SLAs

4.1 Service-Level Indicators

An SLI is a quantitative measure of a service aspect important to users. Examples include successful request proportion, latency below a threshold, freshness of data and the proportion of valid transactions.

For availability measured as successful events divided by valid events:

The numerator and denominator must be defined consistently. For example, planned maintenance, invalid client requests or synthetic health checks may or may not be included, depending on the service contract. The choice must be documented.

4.2 Service-Level Objectives

An SLO is a target value for an SLI over a stated time window. For example, “99.9% of valid checkout requests succeed each calendar month” is an SLO. An SLO is an engineering objective, not a guarantee that every observation will meet the target.

Latency objectives require a distribution rather than an average. A useful objective might be “99% of requests complete within 400 ms.” An average can conceal a small but important population of very slow requests.

4.3 Service-Level Agreements

An SLA is a formal agreement with customers or stakeholders. It may specify targets, measurement methods, support obligations, exclusions and service credits. An SLA is a business and contractual instrument; an SLO is primarily an operational target. Organizations often set internal SLOs stricter than external SLAs to provide a safety margin.

4.4 Error Budgets

If an SLO target is 99.9 percent, the permitted unreliability is 0.1 percent. This permitted amount is the error budget:

For a 30-day month, assuming 30 × 24 × 60 = 43,200 minutes:

This calculation assumes the SLO is availability-based and that all minutes are equally weighted. Request-based SLOs require counting valid requests rather than elapsed time. Error budgets support balanced decisions: when the budget is healthy, teams may accept controlled delivery risk; when it is exhausted, reliability work may take priority.

5. Availability, Reliability and Resilience

Availability is the proportion of time or valid requests during which a service is usable. Reliability concerns consistent successful performance over time. Resilience is the ability to withstand disruption and recover while maintaining an acceptable level of service.

A service can be designed for resilience through:

• redundancy across instances, zones or regions;
• health checks and automated replacement;
• timeouts, retries and circuit breakers;
• idempotent operations;
• queues and asynchronous processing;
• graceful degradation and feature reduction;
• isolation of failure domains;
• tested backups and recovery procedures.

Retries require care. Unbounded or synchronized retries can overload an already failing dependency. Exponential backoff with jitter reduces synchronized demand. A retry is appropriate only when the operation is likely to succeed later and repeating it is safe. Idempotency keys can prevent duplicate effects in operations such as payments.

Redundancy improves availability only when common failure modes are addressed. Two application servers sharing one failed database or power domain do not provide full independence. Architecture should identify dependencies, failure domains and single points of failure.

6. Capacity and Performance

Capacity planning compares expected demand with available sustainable supply. Relevant dimensions include requests per second, concurrent users, storage, memory, network bandwidth, database connections and queue processing rate.

A basic utilization relationship is:

Both quantities must use the same unit and time interval. If a worker pool can process 120 jobs per minute and demand is 90 jobs per minute, utilization is 90 ÷ 120 = 0.75, or 75 percent. Sustained demand above capacity causes backlog or rejected work. Even utilization below 100 percent may produce unacceptable latency because queues grow as the system approaches its limits.

Capacity planning should use historical trends, load tests, growth assumptions and failure scenarios. It should also account for headroom, maintenance, uneven traffic and the capacity lost when a component fails. Autoscaling is useful, but it cannot compensate for slow scaling decisions, an exhausted database or an external dependency with fixed limits.

7. Incident Response

An incident is an event that causes, or risks causing, degradation to a service. Incident response aims to reduce harm quickly, restore service and preserve evidence for learning.

A practical response sequence is:

1. Detect and acknowledge: confirm the signal and establish an incident record.
2. Assess: determine affected services, users, regions, severity and current symptoms.
3. Assign roles: appoint an incident commander, technical lead and communications lead when needed.
4. Contain: stop harmful change, roll back, isolate a component or reduce load.
5. Recover: restore normal service using the safest verified action.
6. Communicate: provide accurate, regular updates to stakeholders.
7. Close and learn: verify recovery, document the timeline and create improvement work.

The incident commander coordinates rather than personally performing every technical task. Clear roles reduce duplicated effort. Severity classifications should be based on user impact, duration, data risk and business criticality, not on which team discovered the issue.

Runbooks should contain prerequisites, diagnostic commands or links, expected observations, decision points, rollback steps and escalation contacts. They must be tested and updated; an untested runbook is only a hypothesis.

8. Post-Incident Reviews

A post-incident review, sometimes called a postmortem, examines what happened, why defenses did not prevent or limit it, and how the organization can improve. A blameless review does not mean ignoring accountability. It means examining decisions and system conditions rather than reducing a complex event to individual fault.

A strong review includes:

• concise impact statement;
• detection and recovery times;
• chronological timeline with evidence;
• contributing technical and organizational conditions;
• what worked and what failed;
• customer and data implications;
• prioritized corrective actions with owners and due dates.

Avoid vague actions such as “be more careful.” Better actions are testable: “Add a deployment check that rejects configurations missing the database timeout,” or “Create an alert for the SLO symptom and test it during the next resilience exercise.” Reviews should distinguish immediate triggers from deeper contributing factors, such as inadequate testing, unclear ownership, unsafe defaults or excessive operational complexity.

9. Disaster Recovery and Business Continuity

Disaster recovery (DR) concerns restoring technology and data after a major disruption. Business continuity is broader: it addresses how critical business functions continue during disruption, including people, facilities, suppliers, communications and manual procedures.

Two central recovery targets are:

• Recovery time objective (RTO): maximum acceptable time to restore a service.
• Recovery point objective (RPO): maximum acceptable amount of data loss measured in time.

For example, an RTO of two hours means the service should be restored within two hours of a qualifying disaster. An RPO of 15 minutes means recovery should lose no more than approximately 15 minutes of committed data, subject to the defined measurement method.

Backup strategies include full, incremental and continuous or replicated approaches. A backup is not a recovery plan until restoration has been tested. Important practices include encryption, access separation, retention policies, immutable or offline copies, geographic diversity and regular restore exercises.

A recovery design may be classified as cold, warm or hot. A cold site requires substantial preparation; a warm site has partially prepared resources; a hot site or active-active arrangement can provide rapid service restoration but is more expensive and operationally complex. The appropriate design depends on business impact, RTO, RPO, cost and technical feasibility.

10. Applied Case: Illustrative Online Retail Incident

Background

An online retailer operates a checkout service composed of a web application, payment service, inventory database and external payment provider. Its monthly availability SLO is 99.9 percent for valid checkout attempts. The team collects application logs, request metrics and distributed traces.

Problem

During a promotional campaign, checkout latency rises and some payments are submitted twice. The dashboard shows elevated latency but no immediate availability alert because most requests eventually succeed. Customer support reports duplicate authorizations.

Analysis

Tracing reveals that the payment provider occasionally responds slowly. The checkout service retries requests after a short timeout. The payment operation is not consistently protected by an idempotency key. Retries increase load and create duplicate payment attempts. The service’s latency SLI does not adequately capture the business risk, and the alert is based on average latency rather than a high percentile or payment-failure symptom.

Approach and Solution

The incident commander pauses the promotional configuration, limits retries with exponential backoff and jitter, and enables a provider-specific circuit breaker. The payment workflow is changed to use idempotency keys, allowing repeated requests to return the original result rather than create a new charge. A customer-facing status page is updated, and affected transactions are reconciled.

After recovery, the team adds an SLI for successful completed checkouts, an SLI for duplicate-payment prevention, and a latency objective based on the 99th percentile. Load testing includes slow-provider scenarios. The payment dependency is documented in the service catalogue, and a runbook explains when to disable the integration.

Outcome and Learning Points

The technical trigger was provider latency, but the incident was amplified by unsafe retries, inadequate business-level indicators and insufficient dependency testing. The key lesson is that resilience mechanisms can become failure multipliers when their assumptions are not tested. User outcomes—not infrastructure metrics alone—must guide reliability design.

11. Governance, Ethics and Contemporary Practice

Observability data may contain personal, confidential or security-sensitive information. Organizations should apply data minimization, access control, encryption, retention limits and audit logging. Logs should not become an uncontrolled secondary database of customer information. Regulatory requirements may govern personal data, financial records, health information and breach notification.

Operational access should follow least privilege. On-call engineers need sufficient access to respond, but emergency access should be time-limited and auditable. Alerting systems should be designed inclusively: excessive night work, unclear escalation and constant interruptions create occupational-health risks and reduce reliability.

Artificial intelligence is increasingly used for anomaly detection, log summarization and incident correlation. These tools can reduce search effort, but they may generate false explanations, conceal uncertainty or expose sensitive telemetry to an external service. AI-generated recommendations require human verification, controlled data handling and auditability. Automation should fail safely and should not silently execute high-impact changes without authorization.

Contemporary practice increasingly emphasizes open telemetry standards, software supply-chain observability, continuous verification, resilience testing and platform engineering. These developments do not remove the need for clear ownership, sound indicators and disciplined incident processes.

Best-Practice Checklist

• Define reliability from the user and business outcome.
• Instrument services with structured logs, useful metrics and trace context.
• Control telemetry cardinality, retention and sensitive content.
• Use dashboards for decisions, not decoration.
• Page only when human action is justified.
• Define SLIs, SLOs and measurement rules explicitly.
• Test backups, failover, runbooks and recovery objectives.
• Use safe retries, timeouts, circuit breakers and idempotency.
• Conduct blameless, evidence-based post-incident reviews.
• Track corrective actions to completion and verify their effect.

Chapter Summary

Operations keeps deployed systems secure, usable and supportable. Observability combines logs, metrics, traces, events and profiles to provide evidence about system behavior. Dashboards organize this evidence for decisions, while alerts identify conditions requiring action. SLIs measure user-relevant service behavior; SLOs establish engineering targets; and SLAs formalize commitments. Error budgets connect reliability with delivery decisions.

Availability, reliability and resilience are related but distinct. Redundancy, isolation, graceful degradation, safe retries and tested recovery procedures help systems withstand failure. Capacity planning compares demand with sustainable supply and must consider headroom and failure scenarios. Effective incident response emphasizes rapid assessment, clear roles, containment, recovery and communication. Post-incident reviews convert experience into system improvement. Disaster recovery uses RTO and RPO to guide backup, restoration and continuity design. Across all these practices, privacy, security, human sustainability and accountable automation are essential.

Key Terms

Observability: The ability to infer internal system state from externally available outputs.

Log: A time-stamped record of an event or state.

Metric: A numerical measurement observed over time.

Trace: A representation of one request’s path through distributed components.

Span: A timed operation within a distributed trace.

Dashboard: A visual display organized to support operational decisions.

Alert: A notification indicating that investigation or action is required.

Service-Level Indicator (SLI): A quantitative measure of a user-relevant service property.

Service-Level Objective (SLO): A target for an SLI over a defined period.

Service-Level Agreement (SLA): A formal service commitment to customers or stakeholders.

Error budget: The permitted unreliability implied by an SLO.

Availability: The proportion of time or valid requests for which a service is usable.

Resilience: The capacity to withstand disruption and recover acceptably.

Incident: An event causing or threatening service degradation.

Runbook: A documented procedure for diagnosing or operating a service.

Post-incident review: A structured examination of an incident and improvement opportunities.

RTO: The maximum acceptable time to restore a service.

RPO: The maximum acceptable period of data loss.

Capacity planning: The process of matching expected demand with sustainable system supply.

Graceful degradation: Continued operation with reduced functionality during partial failure.

Review Questions

Short Answer Questions

1. What is the difference between monitoring and observability?
2. Name the four golden signals.
3. Distinguish an SLI from an SLO.
4. What do RTO and RPO measure?
5. Why can unbounded retries worsen an incident?

Long Answer Questions

1. Explain how logs, metrics and traces complement one another during diagnosis.
2. Describe the characteristics of an effective operational dashboard and alert.
3. Explain how error budgets can influence release and reliability decisions.
4. Discuss the major stages and roles in incident response.
5. Compare cold, warm and hot disaster-recovery arrangements.

Analytical / Application Questions

1. A service has an SLO of 99.95 percent over a 30-day month. Calculate the permitted unavailable time, assuming a time-based SLO and 43,200 minutes in the month. Explain the assumption.
2. A queue processes 600 jobs per hour while demand reaches 540 jobs per hour. Calculate utilization and explain two risks if demand increases to 660 jobs per hour.
3. Design three SLIs for the online retail checkout case. For each, state what user outcome it measures and propose one alerting consideration.

Multiple Choice Questions with Answers

Question 1

Which telemetry type is most suitable for following one request across several distributed services?

A. A trace

B. A gauge metric

C. An infrastructure dashboard

D. A backup record

Correct Answer: A. A trace

Question 2

What is the main purpose of an SLO?

A. To define a target for a service-level indicator

B. To list every server in an organization

C. To replace incident response procedures

D. To specify a programming language

Correct Answer: A. To define a target for a service-level indicator

Question 3

If an availability SLO is 99.9%, what fraction represents the error budget?

A. 0.01%

B. 0.1%

C. 1%

D. 99.9%

Correct Answer: B. 0.1%

Question 4

Which alert is generally most actionable?

A. A page for every debug log

B. A page whenever CPU changes by 1%

C. A page indicating sustained user-impacting checkout failures

D. A page for every successful deployment

Correct Answer: C. A page indicating sustained user-impacting checkout failures

Question 5

Why are idempotency keys useful in payment operations?

A. They increase screen resolution

B. They prevent a repeated request from creating an unintended duplicate effect

C. They eliminate the need for backups

D. They guarantee zero network latency

Correct Answer: B. They prevent a repeated request from creating an unintended duplicate effect

Question 6

What does RPO describe?

A. The maximum acceptable service restoration time

B. The maximum acceptable amount of data loss measured in time

C. The number of responders assigned to an incident

D. The percentage of CPU reserved for monitoring

Correct Answer: B. The maximum acceptable amount of data loss measured in time

Question 7

A worker pool has capacity for 120 jobs per minute and receives 90 jobs per minute. What is its utilization?

A. 25%

B. 50%

C. 75%

D. 133%

Correct Answer: C. 75%

Question 8

What is the primary aim of a blameless post-incident review?

A. To identify a person to punish

B. To remove all operational documentation

C. To understand contributing conditions and improve the system

D. To prevent stakeholders from receiving information

Correct Answer: C. To understand contributing conditions and improve the system

Question 9

Which practice helps reduce synchronized retry overload?

A. Unlimited immediate retries

B. Exponential backoff with jitter

C. Removing all timeouts

D. Retrying failed payments without an identifier

Correct Answer: B. Exponential backoff with jitter

Question 10

Why must backups be restored during testing?

A. A backup is not evidence of recoverability until restoration is demonstrated

B. Restoration testing deletes every production system

C. Backups cannot contain data

D. Testing makes RPO irrelevant

Correct Answer: A. A backup is not evidence of recoverability until restoration is demonstrated