3. Metrics, Time-Series Analysis, and Dashboards

Introduction and Context

Observability begins with the disciplined collection and interpretation of evidence about a system. Logs describe events, traces describe request paths, and metrics provide numerical summaries that can be compared across time. This chapter focuses on metrics as the foundation for operational reasoning: detecting degradation, explaining performance changes, evaluating capacity, and communicating service health.

A metric is useful only when its meaning, scope, sampling method, and limitations are understood. A dashboard displaying hundreds of attractive charts may still fail to answer basic operational questions. Conversely, a small set of carefully designed measurements can reveal whether users are experiencing errors, whether a service is overloaded, and whether an apparent anomaly is genuine or merely seasonal variation.

Metrics are therefore not simply numbers to be collected. They are representations of system behavior. Their design involves choices about aggregation, labels, resolution, retention, statistical summaries, and visual presentation. Poor choices create high cardinality, misleading averages, alert fatigue, and false confidence. Sound choices support reliable engineering and informed management.

Learning Objectives

By the end of this chapter, learners should be able to:

1. Distinguish counters, gauges, histograms, summaries, rates, and derived metrics.

2. Explain labels, cardinality, aggregation, dimensionality, and time-series storage.

3. Apply RED, USE, and golden-signal models to service and infrastructure monitoring.

4. Interpret averages, percentiles, distributions, histograms, and saturation indicators.

5. Analyze trends, seasonality, anomalies, correlations, and causal hypotheses.

6. Design dashboards that support detection, diagnosis, and decision-making.

7. Identify vanity metrics, misleading visualizations, governance risks, and common implementation errors.

Metrics as Operational Evidence

A time series is generally represented as a sequence of timestamped values associated with a metric name and a set of labels. For example:

`http_requests_total{service="checkout", method="POST", status="500"}`

The metric name identifies what is measured; labels provide dimensions for analysis. Each unique combination of metric name and label values constitutes a distinct time series. Labels enable questions such as “Which endpoint is failing?” or “Which region has the highest latency?” However, every additional dimension multiplies the number of series. User identifiers, request IDs, unrestricted URLs, and arbitrary exception messages are usually dangerous labels because they create unbounded or extremely large cardinality.

A good metric has a clear unit, stable semantics, documented source, and a defined interpretation. “CPU usage” might mean host CPU busy time, container CPU quota consumption, or a process-level percentage. These are not interchangeable. Metric names and descriptions should therefore specify whether a value is a count, ratio, duration, bytes, or normalized percentage, and whether it is cumulative or instantaneous.

Principal Metric Types

A counter increases monotonically, except when it is reset, commonly because a process restarts. Examples include total requests, bytes transmitted, and failed transactions. Counters are normally interpreted through a rate over an interval rather than by displaying their absolute value. If a counter rises from 10,000 to 10,600 in five minutes, its average rate is two events per second. Counter resets must be handled by the monitoring system or query logic.

A gauge represents a value that can rise or fall. Temperature, queue depth, memory in use, active sessions, and current replica count are gauges. A gauge’s instantaneous value may be meaningful, but trends and durations are often more useful than a single sample.

A histogram records observations in configurable buckets and usually also maintains a count and sum. It is appropriate for request duration, response size, or queue waiting time because it preserves information about distribution. A summary typically calculates quantiles at the point where observations are collected. Summaries can be convenient, but quantiles from separate instances are difficult to aggregate accurately. Histograms are generally more flexible for fleet-wide analysis because bucket counts can be combined.

A rate is a change in a cumulative quantity over time. An error ratio is commonly calculated as failed requests divided by total requests. A saturation metric expresses how close a resource is to a limit, such as CPU quota, connection-pool capacity, disk space, or queue capacity. Derived metrics must retain a clear relationship to their source measurements; otherwise, users may not know what a percentage represents.

Labels, Aggregation, and Metric Design

Labels support slicing and grouping. A query may calculate the error ratio by service, region, or endpoint. Aggregation should match a real operational question. For example, summing request counts across instances is sensible, while averaging instance-level error percentages can be misleading when instances handle different traffic volumes. Weighted ratios should normally be calculated from aggregated numerators and denominators:

`total errors / total requests`

rather than by averaging per-instance ratios.

The choice of aggregation also depends on the metric type. Counts are often summed, current resource values may be averaged or maximized, and latency is better represented by quantiles or bucket distributions than by a simple sum. Dimensions that are not needed for diagnosis should not be retained indefinitely. A practical design process is to ask: who will use this metric, what decision will it support, what dimensions are required, and what is the expected cardinality?

Metric names and labels should be consistent across services. Standard units, status classifications, and naming conventions reduce cognitive load and permit shared dashboards. Instrumentation should also distinguish business events from technical events. “Orders completed” and “HTTP requests served” may both be valuable, but they answer different questions and must not be substituted for one another.

Three Complementary Monitoring Models

RED for Request-Driven Services

The RED method organizes service-level monitoring around:

• Rate: the number of requests or operations per unit of time.
• Errors: the number or proportion of failed operations.
• Duration: the time required to complete operations, preferably expressed through a distribution or percentiles.

RED is especially effective for APIs, web services, message consumers, and other request-oriented components. A dashboard showing request rate, error ratio, and latency allows an operator to distinguish a traffic surge from a service failure. It does not, however, explain resource exhaustion by itself.

USE for Resources

The USE method examines:

• Utilization: the proportion of a resource’s capacity being used.
• Saturation: the amount of queued work or demand that cannot be immediately served.
• Errors: failures associated with the resource.

For a database connection pool, utilization may be active connections divided by maximum connections, saturation may be the number of waiting requests, and errors may include connection timeouts. For a disk, utilization may refer to busy time, saturation to pending I/O, and errors to read or write failures. USE complements RED by explaining why service behavior may have changed.

The Golden Signals

The golden signals are latency, traffic, errors, and saturation. They provide a broad service-health view and are useful for dashboards and alert design. Latency describes user-visible speed; traffic describes demand; errors describe unsuccessful outcomes; and saturation indicates proximity to a limiting resource. These signals overlap with RED and USE, but the golden-signal model is often applied at the service or system boundary rather than to a particular resource.

These models are frameworks, not substitutes for understanding. A service may show low CPU utilization while suffering from database connection saturation. A low error count may reflect low traffic rather than good health. Metrics must therefore be interpreted together and in context.

Percentiles, Histograms, and the Problem of Averages

The arithmetic mean is easy to calculate but often inadequate for latency. Suppose 99 requests take 100 milliseconds and one request takes 20 seconds. The mean is substantially affected by the outlier, but it does not reveal how many users experienced extreme delay. Conversely, a mean can appear acceptable while a small but important population experiences very poor performance.

A percentile identifies a value below which a specified proportion of observations fall. The 95th percentile latency is the value at or below which approximately 95 percent of requests were completed; the remaining five percent took longer. The 99th percentile is useful for examining tail behavior, especially where slow requests affect user experience or downstream capacity.

Percentiles require careful interpretation. A p99 calculated separately for each instance cannot generally be averaged to obtain a fleet-wide p99. Histograms permit approximate quantile calculation from aggregated bucket counts, provided bucket boundaries are suitable. Bucket design is a trade-off: too few or poorly placed buckets produce imprecise results, while excessive buckets increase storage and processing cost.

Latency distributions may be multimodal. One mode might represent cache hits and another database misses. A single percentile can conceal this structure. Analysts should examine heat maps, bucket distributions, and related dimensions such as endpoint, status, region, and dependency. Percentiles also depend on the selected time window: a daily p99 can hide a brief but severe incident.

Reading Time Series: Trend, Seasonality, and Anomaly

Time-series analysis begins with establishing a baseline. A baseline describes expected behavior for a comparable period and may include typical levels, variation, and known operating conditions. A weekly traffic pattern should not be compared only with the immediately preceding hour. Retail systems may exhibit daily and weekly cycles, while payroll or examination systems may have monthly or academic-calendar effects.

A trend is a persistent directional movement, such as steadily increasing storage consumption. Seasonality is a repeated pattern at a known interval, such as higher traffic during business hours. An anomaly is an observation that differs materially from an expected pattern. An anomaly is not automatically an incident: a planned product launch may create an unusual but legitimate increase in traffic.

Useful analytical techniques include moving averages, rolling percentiles, week-over-week comparison, rate-of-change analysis, and decomposition into baseline and residual components. Smoothing can reveal a trend but may hide short incidents, so raw values should remain available. Dynamic thresholds based on historical variability can be more useful than fixed thresholds for strongly seasonal workloads, but they can also learn abnormal behavior if the baseline is contaminated by incidents.

Correlation is a valuable investigative clue, not proof of causation. A rise in latency and CPU usage may indicate CPU contention, but both may also result from a traffic surge or a dependency failure. Analysts should establish temporal order, control for common factors, compare affected and unaffected groups, and test hypotheses through traces, logs, experiments, or controlled changes.

Dashboards as Instruments for Reasoning

A dashboard should support a defined audience and decision. An on-call dashboard is optimized for rapid detection and triage. A service-owner dashboard may include endpoint-level latency, deployment markers, dependency behavior, and capacity forecasts. An executive dashboard may summarize service-level objectives, availability, customer impact, and trend direction. A single dashboard rarely serves all these purposes well.

A useful dashboard generally follows a progression from outcome to explanation. The first view should show service health and user impact: request volume, error ratio, latency percentiles, and saturation. Lower panels can show resource utilization, dependency metrics, deployment events, and detailed dimensions. Consistent time ranges, units, labels, legends, and annotations help users form an accurate mental model.

Visual encoding matters. A line chart is appropriate for change over time; a stacked area chart can show composition when totals and category relationships are meaningful; a heat map can show latency distributions; and a table can be better than a chart for ranked offenders. Dual axes should be avoided unless the relationship is exceptionally clear, because they can create false visual correlations. Truncated axes, excessive smoothing, inappropriate logarithmic scales, and inconsistent aggregation can exaggerate or minimize changes.

Dashboards should expose data quality. Missing samples, scrape failures, counter resets, and delayed ingestion should be distinguishable from genuine zero values. Every important panel should answer what is measured, over what interval, in what unit, and with what aggregation. Links to traces, logs, runbooks, and deployment history turn a dashboard from a display into an operational tool.

Practical Application: From Questions to Instrumentation

A disciplined monitoring design starts with questions rather than available data. For an online checkout service, relevant questions include: Are customers completing purchases? Is demand changing? Which operations fail? Is latency concentrated in the tail? Is the service constrained by compute, storage, or dependencies?

These questions lead to measurements such as completed and failed checkout attempts, request rate by operation, p95 and p99 duration, payment-provider error ratio, queue depth, database connection saturation, and deployment version. A dashboard can place business outcomes beside technical signals, allowing the team to distinguish “few orders because demand is low” from “few orders because checkout is failing.”

Alerts should be tied to user impact or a meaningful risk. An alert on a single CPU threshold may generate noise during harmless traffic bursts, while an alert on a sustained error-budget burn or elevated checkout failure ratio is more actionable. Alert thresholds should account for duration, traffic volume, and seasonality. Low-volume services require special care because a small number of failures can produce a dramatic percentage without representing a statistically reliable change.

Illustrative Case Study: Diagnosing a Checkout Latency Incident

Background

A fictional online retailer operates a checkout API across three regions. The team’s dashboard reports average latency, CPU utilization, request count, and HTTP 5xx count. The service has recently introduced a recommendation lookup during checkout.

Situation

Customer-support reports indicate that checkout is intermittently slow. The dashboard shows average latency of 420 milliseconds, only slightly above the normal 350 milliseconds, and CPU utilization below 60 percent. No CPU alert has fired. Engineers initially conclude that the issue is minor.

Analysis

A review of request-duration histograms shows that the p50 remains near 180 milliseconds, but p99 has increased from 900 milliseconds to 4.8 seconds. The average concealed a small but important tail. Grouping by endpoint reveals that the recommendation-enhanced checkout path is affected. Grouping by region shows that one region has substantially higher latency.

The RED view identifies a stable request rate, a modest increase in errors, and severe tail-latency degradation. The USE view for the application nodes appears normal, but database connection-pool saturation in the affected region is high. Trace data shows that recommendation requests are waiting for a slow database query, holding checkout connections longer than before.

Approach and Solution

The team temporarily disables recommendations in the checkout path, reducing database demand and restoring p99 latency. Engineers then optimize the query, add an appropriate index, and introduce a bounded timeout and fallback behavior. They revise instrumentation to record checkout latency by feature flag, dependency duration, region, and outcome. The dashboard is redesigned to show p50, p95, and p99 latency, connection-pool saturation, dependency duration, and completed-checkout ratio.

Outcome

The incident becomes detectable through tail latency and saturation before widespread errors occur. The revised dashboard also distinguishes the customer-facing outcome from infrastructure activity. The team adds a release annotation and a runbook link so that future regressions can be investigated more quickly.

Learning Points

The case demonstrates that averages can hide user harm, low CPU does not prove that a service has spare capacity, and dependency saturation may be the limiting factor. It also shows why dashboards should connect RED signals with USE signals and business outcomes. Instrumentation becomes more valuable when it supports a specific decision rather than merely increasing data volume.

Common Errors, Risks, and Governance Considerations

One common mistake is collecting vanity metrics: values that look impressive but do not support a decision. Total uptime without traffic context, cumulative request counts without rates, or the number of deployed services without service-quality measures may be interesting but operationally weak. Another mistake is using a single aggregate that hides important populations, such as a global average across regions or endpoints.

High-cardinality labels can increase storage cost and impair query performance. Metrics may also expose sensitive information if labels contain customer identifiers, account names, locations, or request content. Monitoring data should follow access-control, retention, encryption, and data-minimization policies. Even apparently technical telemetry can become personal data when it is linkable to individuals or behavior.

Metric definitions require governance. Teams should document ownership, units, collection intervals, retention, alert significance, and known failure modes. Changes to instrumentation should be reviewed like code changes because a renamed metric, changed bucket boundary, or altered error classification can invalidate dashboards and historical comparisons.

Automation and machine learning can assist anomaly detection, capacity forecasting, and correlation discovery. Such systems should not be treated as unquestionable authorities. Models may learn incident conditions as normal, produce alerts without actionable explanations, or amplify bias in incomplete data. Human review, transparent baselines, and explicit fallback procedures remain essential.

Current Developments and Best Practices

Modern observability platforms increasingly combine metrics, logs, and traces through shared resource attributes such as service name, environment, version, and region. Open standards for telemetry improve portability, while exemplars can connect a metric observation to a representative trace. These capabilities reduce the distance between detection and diagnosis.

Effective practice includes the following:

• Define service-level indicators around user-relevant outcomes.
• Use counters for events, gauges for current state, and histograms for distributions.
• Calculate fleet-wide ratios from aggregated numerators and denominators.
• Prefer percentiles and distributions to averages for latency analysis.
• Control label cardinality and exclude unbounded identifiers.
• Design dashboards for a stated audience and operational decision.
• Include deployment annotations, data-quality indicators, and links to investigation tools.
• Review alerts for actionability, duration, traffic volume, and seasonality.
• Test dashboards and alert rules during controlled failure exercises.
• Retire metrics that no longer support a decision.

Chapter Summary

Metrics convert system behavior into time-indexed evidence, but their value depends on precise semantics, suitable aggregation, and appropriate interpretation. Counters, gauges, histograms, and summaries serve different measurement purposes. Labels enable analysis but create cardinality and governance risks. RED emphasizes rate, errors, and duration for services; USE examines utilization, saturation, and errors for resources; and the golden signals combine latency, traffic, errors, and saturation into a broad health model.

Averages can conceal tail latency, making percentiles and histograms essential for performance analysis. Time-series interpretation requires attention to trends, seasonality, baselines, anomalies, and the difference between correlation and causation. Dashboards should be designed as instruments for decisions, beginning with user outcomes and progressing toward explanatory signals. Professional monitoring avoids vanity metrics, misleading graphics, uncontrolled dimensions, and unactionable alerts. The strongest practice combines technical telemetry, business context, governance, and disciplined investigation.

Key Terms

Metric: A quantitative representation of an observed system or business property.

Time series: Timestamped observations of a metric, commonly identified by name and labels.

Counter: A monotonically increasing measurement used for cumulative events.

Gauge: A measurement representing a value that may increase or decrease.

Histogram: A distribution-oriented metric using buckets, count, and sum.

Summary: A metric that reports statistical summaries, often quantiles, at collection time.

Label: A named dimension used to identify and group metric series.

Cardinality: The number of distinct label combinations or time series.

Rate: The change in a cumulative quantity per unit of time.

Error ratio: Failed operations divided by total operations.

Percentile: A value below which a specified proportion of observations lies.

Tail latency: The slower portion of a latency distribution, often represented by p95 or p99.

RED: Rate, Errors, and Duration monitoring model for request-driven services.

USE: Utilization, Saturation, and Errors model for resources.

Golden signals: Latency, traffic, errors, and saturation indicators of service health.

Saturation: The degree to which demand approaches or exceeds available capacity.

Seasonality: A recurring time-based pattern in a time series.

Anomaly: An observation that differs materially from an expected baseline.

Vanity metric: A measure that appears informative but does not support a meaningful decision.

High cardinality: A condition in which metric dimensions generate very many distinct series.

Review Questions

Short Answer Questions

1. Why are counters usually interpreted through rates rather than absolute values?

2. Distinguish a gauge from a histogram and give one example of each.

3. What is metric cardinality, and why can it become an operational problem?

4. Why can an average latency be misleading?

5. State the three elements of the RED method.

Long Answer Questions

1. Explain how labels and aggregation affect the accuracy, cost, and usefulness of a metric system.

2. Compare RED, USE, and the golden-signal model, including the situations in which each is most useful.

3. Discuss the advantages and limitations of histograms and summaries for fleet-wide percentile analysis.

4. Explain how trend, seasonality, baseline, anomaly, and correlation should be distinguished in time-series analysis.

5. Describe the principles of designing an operational dashboard for an on-call engineering team.

Analytical / Application Questions

1. A service has stable average latency but a rapidly rising p99. What additional metrics and dimensions would you examine, and why?

2. Three instances report error ratios of 1%, 5%, and 10%, but they handle 10,000, 1,000, and 100 requests respectively. Explain why averaging the ratios is inappropriate and describe the correct calculation approach.

3. A dashboard shows rising CPU, rising latency, and rising traffic after a deployment. Construct at least two competing causal hypotheses and identify evidence that would distinguish them.

22. Multiple Choice Questions with Answers

Question 1

Which metric type is most appropriate for representing request duration distributions that must be aggregated across service instances?

A. Gauge

B. Counter

C. Histogram

D. Boolean flag

Correct Answer: C. Histogram

Question 2

A service receives 10,000 requests and records 200 failures. What is the error ratio?

A. 0.2%

B. 2%

C. 20%

D. 98%

Correct Answer: B. 2%

Question 3

Which RED signal is most directly represented by requests per second?

A. Rate

B. Errors

C. Duration

D. Saturation

Correct Answer: A. Rate

Question 4

Why should user IDs generally not be used as metric labels?

A. They prevent counters from increasing

B. They create potentially unbounded cardinality

C. They make gauges negative

D. They eliminate time-series data

Correct Answer: B. They create potentially unbounded cardinality

Question 5

A p99 latency rises sharply while p50 remains stable. What is the most likely interpretation?

A. All requests became equally faster

B. The slow tail has degraded while typical requests remain similar

C. The request rate must have fallen to zero

D. The metric cannot contain any outliers

Correct Answer: B. The slow tail has degraded while typical requests remain similar

Question 6

Which calculation is generally correct for a fleet-wide error ratio?

A. Average the error percentages of all instances

B. Use the highest instance percentage only

C. Sum errors and divide by summed requests

D. Divide total requests by total errors

Correct Answer: C. Sum errors and divide by summed requests

Question 7

In the USE method, a growing queue of work waiting for a resource primarily indicates:

A. Utilization only

B. Saturation

C. Availability only

D. Cardinality

Correct Answer: B. Saturation

Question 8

Which dashboard design choice is most likely to create a misleading visual comparison?

A. Adding deployment annotations

B. Showing units in panel titles

C. Using unrelated dual axes without clear explanation

D. Linking a metric to a trace

Correct Answer: C. Using unrelated dual axes without clear explanation

Question 9

A traffic metric increases every weekday between 9 a.m. and 5 p.m. This repeated pattern is best described as:

A. Seasonality

B. Cardinality

C. Counter reset

D. Random noise

Correct Answer: A. Seasonality

Question 10

Why is correlation between CPU and latency insufficient to establish that CPU caused the latency increase?

A. Metrics cannot have timestamps

B. Both may be consequences of another factor, such as increased traffic

C. CPU is never related to performance

D. Latency cannot be measured with a histogram

Correct Answer: B. Both may be consequences of another factor, such as increased traffic