5. Observability, Telemetry and Reliability Monitoring

Introduction and Context

Site Reliability Engineering treats reliability as an engineering property that must be measured, analysed, and improved. A service cannot be operated responsibly when its internal condition is invisible. When users report that an application is slow, an operations team should be able to determine which service is affected, how many users are experiencing the problem, when it began, and whether the cause lies in application code, infrastructure, dependencies, configuration, or demand. Observability provides the technical and analytical means to answer these questions.

Monitoring and observability are related but not identical. Monitoring generally evaluates known conditions through predefined measurements and alerts. Observability is the broader capability of inferring the internal state of a complex system from the data it emits. Monitoring may tell an engineer that error rate has exceeded a threshold; observability helps explain which requests, users, deployments, dependencies, or resource conditions produced that result. Effective SRE practice requires both: monitoring for dependable detection and observability for investigation and learning.

This chapter examines metrics, logs, traces, events, and profiles; the instrumentation and telemetry pipelines that produce them; and the design of dashboards and alerts that support fast, rational decisions. It also addresses cardinality, retention, cost, privacy, alert fatigue, and governance. These are not merely tooling concerns. They influence incident response, service-level objectives, architecture, team behaviour, and the economic sustainability of operating software.

Learning Objectives

After completing this chapter, the learner should be able to:

1. Distinguish monitoring, observability, telemetry, and debugging.

2. Explain the roles and limitations of metrics, logs, traces, events, and profiles.

3. Design instrumentation and a telemetry pipeline for distributed services.

4. Apply RED, USE, and service-level-indicator approaches to reliability monitoring.

5. Create dashboards and symptom-based alerts aligned with user impact and SLOs.

6. Control noise, cardinality, retention, sampling, privacy risk, and observability cost.

7. Evaluate observability architectures and make appropriate professional and managerial decisions.

From System Signals to Operational Understanding

Telemetry is data emitted by a system about its execution and condition. The conventional signal categories are metrics, logs, traces, events, and profiles. No single signal is sufficient in every situation; their value increases when they can be correlated through shared attributes such as timestamp, service, deployment version, request identifier, and trace identifier.

A metric is a numerical observation recorded over time. Examples include request rate, error count, latency, queue depth, CPU utilisation, and the number of active sessions. Metrics are efficient for aggregation, trend analysis, and alerting. A time-series database can store and query them rapidly. Their weakness is that aggregation can hide individual context. An average latency may appear healthy while a small but important group of requests experiences severe delay.

A log is a time-stamped record of an event or state observation. Logs may describe a failed authentication attempt, a configuration change, a database timeout, or a completed business operation. Structured logs, usually encoded as JSON or another machine-readable format, are more useful than free-form text because fields can be queried consistently. Logs should communicate what happened, where it happened, and the relevant context without exposing passwords, tokens, or unnecessary personal information.

A trace represents the path of a request or transaction through one or more components. A distributed trace is composed of spans, each representing an operation such as an HTTP request, database query, message publication, or remote procedure call. Traces are especially valuable for identifying critical paths, dependency latency, retries, and fan-out behaviour. They are less suitable than metrics for broad, inexpensive long-term trend monitoring.

An event records a significant state transition or occurrence, such as a deployment, feature-flag change, node replacement, certificate renewal, or scaling action. Events provide temporal context. A rise in error rate immediately after a deployment is more actionable when the deployment is visible on the same dashboard. Events are not limited to failures; normal operational changes can explain apparent anomalies.

A profile describes resource consumption in a running program, commonly including CPU time, memory allocation, lock contention, or input/output activity. Continuous or periodic profiling can reveal inefficient functions that conventional request metrics cannot identify. Profiling is particularly useful when latency or resource use deteriorates without an obvious increase in error rate. Because profiles may capture code paths or sensitive execution details, access and retention require care.

The practical distinction between monitoring and observability can be expressed through the questions each supports. Monitoring asks, “Is a known condition within an acceptable range?” Observability asks, “What internal conditions could explain this previously unknown or unexpected behaviour?” Monitoring is usually designed in advance around anticipated failure modes. Observability supports exploration, hypothesis formation, and investigation. It does not eliminate the need for instrumentation design; poorly selected or missing telemetry cannot be recovered by a sophisticated query interface.

Debugging is a related activity rather than a signal type. It is the process of locating and explaining a defect or undesirable behaviour. Observability shortens debugging by supplying evidence, but it does not replace sound reasoning, controlled experiments, code review, or testing.

Reliability Models for Measurement

A useful observability design begins with the user and the service objective rather than with a vendor dashboard. The first question is what the service promises: successful checkout, timely API responses, durable message processing, or availability of a reporting function. That promise is translated into service-level indicators (SLIs), targets called service-level objectives (SLOs), and, where relevant, contractual service-level agreements (SLAs).

An SLI is a quantitative measure of service performance. For a request-based API, availability may be defined as:

Availability SLI = successful valid requests / total valid requests

A latency SLI may be expressed as the proportion of valid requests completed below a specified threshold. The SLO might require 99.9 percent of valid requests to succeed over a 30-day window. The SLO should reflect user experience, not merely internal resource health. CPU utilisation is operationally important, but it is not automatically a reliability SLI.

The RED method is useful for request-driven services:

• Rate: the number of requests or operations over time.
• Errors: the number or proportion of failed operations.
• Duration: the time required to complete operations.

The USE method is useful for resources such as hosts, databases, queues, and storage:

• Utilisation: the proportion of a resource being used.
• Saturation: the degree to which work is delayed because capacity is constrained.
• Errors: faults associated with the resource.

RED and USE are complementary. RED describes service behaviour from the request perspective; USE describes resource conditions that may contribute to that behaviour. Neither should be applied mechanically. A queue-based system may require age of oldest message and processing lag as primary indicators, while a batch system may need completion timeliness and data-quality measures.

A strong alert generally follows the symptom principle: alert on a condition that threatens a user-facing objective or requires human action. An alert for high CPU alone may be informative but not urgent if the service remains healthy and capacity is available. An alert for a projected SLO violation, sustained elevated error rate, or growing queue age is more directly connected to reliability. Cause-oriented signals remain useful for diagnosis and dashboards, but they should not generate unnecessary pages.

Error budgets make the connection between measurement and decision-making explicit. If an SLO permits 0.1 percent unavailability over a period, the corresponding error budget is the permitted amount of unsuccessful service. Consuming the budget rapidly may justify slowing feature delivery, increasing testing, prioritising reliability work, or investigating a risky dependency. Observability therefore supports governance and prioritisation, not only incident response.

Instrumentation and Telemetry Architecture

Instrumentation is the deliberate insertion or activation of measurement mechanisms in application and infrastructure components. It may be manual, automatic, or hybrid. Automatic instrumentation can capture common framework operations such as HTTP requests and database calls with limited code changes. Manual instrumentation is needed for business operations, custom queues, meaningful domain attributes, and important boundaries that automatic agents cannot understand.

Instrumentation should establish a consistent semantic model. Request counters should distinguish success and failure in a stable way; latency measurements should use appropriate units; logs should have predictable field names; and traces should carry context across process and asynchronous boundaries. Distributed context propagation allows a request identifier or trace identifier to connect activity across services. Without propagation, engineers see isolated fragments rather than a transaction.

A typical telemetry architecture contains five stages:

1. Production: applications, hosts, containers, networks, databases, and platforms emit signals.

2. Collection: agents, libraries, or collectors receive and enrich telemetry.

3. Processing: data is parsed, sampled, filtered, transformed, redacted, and routed.

4. Storage and indexing: backends retain time series, logs, traces, profiles, and events according to different access patterns.

5. Analysis and action: dashboards, queries, alerts, incident tools, and automated responses consume the data.

A collector layer can decouple applications from particular storage vendors. It can add environment and ownership metadata, enforce sampling policy, remove sensitive fields, and route high-value data to different destinations. This separation improves portability and governance, although it introduces operational components that themselves require monitoring.

Metric design deserves particular discipline. Counters should normally increase monotonically and be converted into rates for interpretation. Gauges represent values that may rise or fall, such as memory usage or queue depth. Histograms record distributions and allow percentile analysis; they are generally preferable to reporting only averages when latency variability matters. Summaries may calculate quantiles at the client but can be difficult to aggregate accurately across instances.

Labels or dimensions make metrics powerful but create cardinality. Cardinality is the number of distinct combinations of label values. Labels such as method, route template, status class, and service are usually manageable. Labels containing user ID, request ID, arbitrary URL, or unbounded exception text can create millions of time series. High cardinality increases memory, storage, query time, and cost, and can make a monitoring system itself unreliable. A practical rule is to use bounded, analytically meaningful dimensions in metrics and retain high-detail identifiers in sampled traces or searchable logs where appropriate.

Sampling is a controlled reduction of telemetry volume. Head-based sampling decides at the beginning of a trace whether to retain it. Tail-based sampling can retain traces after observing their outcome, making it more likely to preserve errors, high latency, or unusual behaviour. Sampling policy should protect representative successful traffic as well as failures; otherwise operators may misunderstand normal performance. Critical audit or security records may require complete retention and should not be treated as ordinary diagnostic telemetry.

Retention should reflect use. High-resolution metrics may be retained for shorter periods and downsampled for longer-term capacity analysis. Detailed logs and traces may have shorter retention than aggregated SLO data. Retention decisions should consider incident investigation windows, compliance obligations, storage cost, and the sensitivity of the data. “Keep everything forever” is rarely a defensible operational or privacy policy.

Dashboards That Support Decisions

A dashboard is an interface for reasoning, not a decorative collection of graphs. Its design should begin with a question and an audience. An executive reliability view may show SLO attainment, error-budget consumption, major incidents, and service dependencies. An on-call overview may show request rate, error ratio, latency percentiles, saturation, active alerts, and recent deployments. A service owner may require endpoint, tenant, version, and dependency breakdowns.

A useful dashboard follows a progressive structure. The first row presents user-facing health and SLO status. The next level provides service signals using RED or an equivalent model. Subsequent panels support diagnosis through dependency latency, resource saturation, queue conditions, logs, traces, and correlated events. Links should allow an operator to move from an aggregate symptom to a narrower view without manually reconstructing query parameters.

Charts must be interpretable. Units, time zones, aggregation methods, percentile definitions, and missing-data behaviour should be explicit. A graph that combines averages from one service with percentiles from another can lead to false comparisons. The dashboard should distinguish no data from zero and should identify deployments or configuration changes on relevant time axes.

High-quality dashboards are intentionally limited. Displaying every available metric creates visual competition and slows recognition. A dashboard should answer the likely first questions during an incident: Is the user impact real? When did it begin? Which operation or population is affected? Is the problem regional, version-specific, or dependency-specific? Is the system recovering or deteriorating?

Alerting as a Human-Factors Discipline

An alert is a request for attention, usually with an implied response. Alerts should therefore have an owner, a severity, a clear condition, a link to supporting evidence, and an indication of expected action. A page should normally represent a situation that an on-call engineer must investigate promptly. A ticket or dashboard annotation may be more appropriate for a condition that can be handled during working hours. Informational telemetry need not become an alert.

Threshold alerts are simple but can be brittle. A fixed latency threshold may page repeatedly during predictable traffic variation, while a static CPU threshold may miss a gradual capacity problem. Rate-of-change, anomaly, burn-rate, and multi-window approaches can improve sensitivity. SLO burn-rate alerting compares the rate at which error budget is being consumed with the rate that would exhaust it within a chosen period. Combining a short window for rapid detection with a longer window for confirmation can reduce transient noise.

Alert quality can be evaluated through actionable rate, false-positive rate, missed incidents, time to acknowledge, and time to mitigate. Alert fatigue occurs when engineers receive too many low-value notifications and begin ignoring or suppressing them. Deduplication, grouping, inhibition of dependent alerts, maintenance windows, and carefully defined escalation policies help control noise. However, suppressing alerts without improving underlying detection merely hides risk.

Applied Case: An Illustrative Retail Checkout Incident

Background

A fictional online retailer operates a checkout platform composed of a web frontend, an order service, a payment adapter, and an inventory service. The company has an SLO requiring 99.95 percent of valid checkout requests to succeed over a rolling 30-day period. Its observability platform collects RED metrics, structured logs, distributed traces, deployment events, and infrastructure metrics.

Situation and Problem

The on-call engineer receives a page indicating that the checkout error-budget burn rate is high. The homepage remains available, and host CPU is normal. A previous monitoring design would have treated the incident as a database-capacity problem because database connection utilisation is elevated. The engineer must identify the user-facing failure and determine whether rollback or dependency intervention is appropriate.

Analysis

The service dashboard shows normal checkout request volume but an increase in failed requests. The latency percentile for checkout has also risen, while unrelated browsing requests remain healthy. Breaking down the metric by route template and status class shows that failures are concentrated in payment confirmation. A deployment event appears eight minutes before the increase.

The engineer selects a failed trace. The trace crosses the order service and payment adapter; the payment call contains repeated retries, each consuming part of the request deadline. Structured logs show a new payment-provider response code that the adapter classifies incorrectly. The database connection increase is a secondary effect: requests remain open during retries and hold connections longer. This illustrates why a resource symptom alone did not identify the primary cause.

Approach and Solution

The incident commander coordinates a rollback of the payment-adapter deployment while a developer verifies the response-code mapping. The team temporarily reduces retry attempts for the affected response and confirms that successful checkout rates recover. The dashboard records the deployment and rollback events, and a representative failed trace is attached to the incident record.

After recovery, the team adds a metric for payment-provider outcome classes, creates a test using recorded provider responses, and revises the alert to page on checkout SLO burn rather than database connection utilisation. A diagnostic alert for unusual retry volume remains visible to the service owner but does not page independently when checkout is healthy.

Outcome and Learning Points

The incident demonstrates the difference between cause-oriented and symptom-oriented monitoring. The user-facing SLO identified the true impact; traces connected the impact to a dependency call; logs explained the unexpected response; and events provided deployment context. The case also shows that correlated signals are more valuable than isolated dashboards and that retries can transform a dependency defect into resource saturation.

Challenges, Risks, and Professional Governance

Observability systems can become expensive and difficult to govern. Excessive detail increases ingestion, indexing, storage, and query costs. Teams may respond by reducing collection indiscriminately, thereby losing evidence needed for incidents. A better approach classifies telemetry by purpose and value: SLO metrics for durable reliability analysis, diagnostic data for investigation, security records for controlled audit, and short-lived high-volume data for active troubleshooting.

Privacy is a central concern. URLs, query parameters, log messages, traces, and profiles may contain names, account identifiers, location information, tokens, or financial data. Instrumentation must apply data minimisation, redaction, access control, encryption, retention limits, and auditability. Sensitive values should not be copied into labels, because metric labels are often widely replicated and difficult to delete. Governance must also address cross-border data transfer and applicable organisational or regulatory requirements.

Common mistakes include logging only failures, treating every log line as an alert, using averages for tail latency, naming metrics inconsistently, attaching unbounded labels, failing to propagate trace context, and creating dashboards without owners. Another mistake is measuring infrastructure because it is easy while neglecting business transactions. Observability should be designed with service owners, developers, security professionals, and operations personnel rather than delegated entirely to a tooling team.

Telemetry pipelines are themselves production systems. Collectors can drop data, queues can fill, agents can consume excessive resources, and backends can become unavailable. Teams should monitor collection lag, dropped spans, ingestion errors, storage capacity, query latency, and cost. A monitoring system that fails silently during an incident creates dangerous confidence.

Automation, Analytics, and Emerging Practice

Modern platforms increasingly use standardised instrumentation libraries and vendor-neutral telemetry protocols. This improves portability and encourages common naming and context-propagation practices. Continuous profiling is becoming more practical because it can reveal resource-intensive code in production with lower overhead than traditional diagnostic sessions.

Machine-learning techniques can assist anomaly detection, event correlation, and probable-cause analysis. Their usefulness depends on data quality and explainability. An anomaly detector that pages whenever traffic changes may produce noise, while a model that cannot explain its evidence may weaken operator trust. AI-assisted query generation and incident summarisation can accelerate investigation, but generated interpretations must be verified against raw telemetry. Automation should support, not replace, accountable engineering judgement.

OpenTelemetry-style approaches illustrate a broader movement toward common instrumentation and collection standards. Standardisation can reduce vendor lock-in, but it does not solve semantic ambiguity, poor sampling decisions, or excessive cardinality. The enduring design questions remain: what user outcome is being measured, what decision will the signal support, who owns it, and what is the cost of retaining it?

Best-Practice Design Principles

A mature observability programme generally follows these principles:

• Begin with user journeys, SLIs, SLOs, and error budgets.
• Use metrics for aggregation and alerting, logs for event detail, traces for request paths, profiles for resource attribution, and events for operational context.
• Prefer structured, consistently named telemetry with propagated context.
• Alert on actionable symptoms and use diagnostic signals to explain them.
• Control cardinality through bounded dimensions and deliberate data classification.
• Apply sampling, retention, redaction, and access controls according to purpose and risk.
• Design dashboards for progressive investigation rather than visual completeness.
• Test alert rules and telemetry pipelines during game days and controlled failures.
• Assign ownership for every important dashboard, alert, and data policy.
• Review signal quality, incident usefulness, cost, and privacy impact continuously.

Chapter Summary

Observability enables engineers to infer system behaviour from emitted telemetry, while monitoring evaluates known conditions through predefined measurements. Metrics, logs, traces, events, and profiles provide complementary evidence. Metrics support efficient aggregation and SLO alerting; logs preserve structured event detail; traces reveal distributed request paths; events establish operational context; and profiles identify code-level resource consumption.

Effective observability begins with user-facing SLIs and SLOs and uses models such as RED and USE to organise service and resource signals. Instrumentation must propagate context and use consistent semantics. Telemetry pipelines collect, process, sample, redact, route, and retain data, but they also require monitoring themselves. Dashboards should guide decisions, and alerts should identify actionable symptoms rather than merely report every abnormal resource measurement. Cardinality, cost, retention, privacy, and alert fatigue are essential design constraints. In mature SRE practice, observability is simultaneously a technical capability, an operational discipline, and a governance responsibility.

Key Terms

Observability: The ability to infer a system’s internal state from its externally emitted data.

Telemetry: Data emitted by software or infrastructure about execution, condition, or change.

Metric: A numerical measurement recorded over time.

Log: A time-stamped record describing an event, action, or state observation.

Trace: A representation of a request’s path through one or more system components.

Span: A timed operation within a distributed trace.

Event: A record of a significant operational or system state change.

Profile: A statistical description of program resource consumption or execution behaviour.

Instrumentation: Code, libraries, agents, or mechanisms that generate telemetry.

SLI: A quantitative indicator of a service characteristic such as success or latency.

SLO: A target value or range established for an SLI.

Error budget: The amount of unreliability permitted by an SLO.

RED method: Monitoring approach based on rate, errors, and duration.

USE method: Resource-oriented approach based on utilisation, saturation, and errors.

Cardinality: The number of distinct combinations of metric dimensions or label values.

Sampling: Selective retention of telemetry to control volume and cost.

Burn rate: The speed at which an error budget is being consumed.

Alert fatigue: Reduced responsiveness caused by excessive or low-value alerts.

Context propagation: Transmission of identifiers and metadata across service boundaries.

Review Questions

Short Answer Questions

1. How does observability differ from conventional monitoring?

2. What information does a distributed trace provide that a metric may not?

3. Define metric cardinality and explain why it affects reliability.

4. What are the three components of the RED method?

5. Why should alerts generally be based on actionable symptoms?

Long Answer Questions

1. Explain the roles and limitations of metrics, logs, traces, events, and profiles in an SRE observability strategy.

2. Describe a telemetry pipeline from instrumentation through collection, processing, storage, and analysis.

3. Discuss how SLIs, SLOs, and error budgets influence dashboard and alert design.

4. Compare the RED and USE methods and explain when each is most appropriate.

5. Analyse the privacy, cost, retention, and cardinality trade-offs in designing an observability platform.

Analytical / Application Questions

1. An API has normal CPU utilisation but its 99th-percentile latency and failed-request ratio have increased after a deployment. Design an investigation using metrics, traces, logs, and events.

2. A team proposes adding user ID, request ID, and full URL as metric labels to improve debugging. Evaluate the proposal and recommend a safer telemetry design.

3. An organisation receives hundreds of pages during every traffic spike, although users experience no measurable SLO violation. Redesign its alerting and dashboard approach, explaining how you would validate the improvement.

22. Multiple Choice Questions with Answers

Question 1

Which statement best distinguishes observability from monitoring?

A. Observability stores only infrastructure metrics, whereas monitoring stores application logs.

B. Monitoring evaluates known conditions, while observability supports inference about internal state from emitted evidence.

C. Monitoring is used during incidents, whereas observability is used only during development.

D. Observability eliminates the need for alerts and service-level objectives.

Correct Answer: B. Monitoring evaluates known conditions, while observability supports inference about internal state from emitted evidence.

Question 2

Which signal is most directly suited to showing the path of a request through several microservices?

A. Gauge metric

B. Structured log only

C. Distributed trace

D. CPU profile

Correct Answer: C. Distributed trace

Question 3

A team wants to alert on a user-facing checkout failure that may be caused by several different components. Which approach is most appropriate?

A. Page whenever any host exceeds 70 percent CPU.

B. Page on checkout SLO burn or sustained checkout error rate.

C. Page on every warning-level log message.

D. Page whenever database storage is above 50 percent.

Correct Answer: B. Page on checkout SLO burn or sustained checkout error rate.

Question 4

Which label is most likely to create dangerous metric cardinality?

A. HTTP method

B. Status class

C. Route template

D. User identifier

Correct Answer: D. User identifier

Question 5

In the RED method, “D” refers to:

A. Dependency count

B. Duration

C. Data loss

D. Deployment frequency

Correct Answer: B. Duration

Question 6

What is a principal advantage of tail-based trace sampling?

A. It guarantees that every successful request is retained.

B. It can retain traces selectively after observing outcomes such as errors or high latency.

C. It removes the need for context propagation.

D. It prevents all telemetry storage costs.

Correct Answer: B. It can retain traces selectively after observing outcomes such as errors or high latency.

Question 7

During an incident, a dashboard shows normal host CPU but elevated checkout latency and repeated payment calls. Which signal would best help identify the causal request path?

A. A distributed trace with payment-adapter spans

B. A monthly storage-capacity chart

C. An unrelated frontend gauge

D. A static list of deployment owners

Correct Answer: A. A distributed trace with payment-adapter spans

Question 8

Why are averages often insufficient for latency monitoring?

A. Averages cannot be stored in time-series systems.

B. Averages always exaggerate the slowest request.

C. Averages can hide poor tail performance affecting a smaller population of users.

D. Averages measure only infrastructure and never application behaviour.

Correct Answer: C. Averages can hide poor tail performance affecting a smaller population of users.

Question 9

Which practice best protects sensitive information in telemetry?

A. Place tokens in metric labels so they are easy to search.

B. Retain all logs indefinitely for investigative flexibility.

C. Apply minimisation, redaction, access control, and purpose-based retention.

D. Disable all application logging.

Correct Answer: C. Apply minimisation, redaction, access control, and purpose-based retention.

Question 10

A monitoring backend begins dropping telemetry during high traffic. What is the most appropriate conclusion?

A. The telemetry platform is not part of the production reliability boundary.

B. Only application errors need investigation; dropped telemetry is harmless.

C. The collection and storage pipeline requires its own reliability monitoring.

D. The team should add more unbounded labels to compensate.

Correct Answer: C. The collection and storage pipeline requires its own reliability monitoring.