Performance, Reliability and Observability

Introduction

A web application is successful only when it delivers correct functionality quickly, consistently, and measurably. A feature that works in a developer’s local environment may still fail under real-world conditions: users may have slow networks, inexpensive mobile devices, high latency to the server, expired sessions, overloaded databases, or browsers with limited memory. Web performance and reliability therefore form an engineering discipline rather than a final-stage optimisation activity.

Performance concerns how efficiently a system responds and uses resources. Reliability concerns its ability to perform correctly over time and under expected variation or failure. Observability concerns how well engineers can infer the internal state of a system from its outputs, especially logs, metrics, traces, and user-facing error reports. These areas are closely related. Poor database performance may cause timeouts; timeouts may generate retries; retries may overload the system; and inadequate telemetry may prevent the cause from being discovered.

This chapter presents an integrated approach to measuring and improving browser rendering, network delivery, caching, images, JavaScript bundles, database queries, logging, monitoring, tracing, error reporting, scalability, and reliability.

Learning Objectives

By the end of this chapter, students should be able to:

1. Explain the main stages of browser rendering and identify their performance costs.
2. Interpret Core Web Vitals and select suitable diagnostic tools.
3. Improve delivery through caching, compression, image optimisation, and code splitting.
4. Analyse database queries and distinguish latency, throughput, and resource utilisation.
5. Design useful logs, metrics, traces, and error-reporting workflows.
6. Apply reliability principles including timeouts, retries, graceful degradation, and capacity planning.
7. Construct a performance and reliability improvement plan using evidence rather than intuition.

1. A Systems View of Web Performance

A user request commonly passes through several stages:

1. The browser resolves a domain name using DNS.
2. It establishes a connection, often using TCP and TLS, or QUIC in HTTP/3.
3. It requests documents and assets from a server or content delivery network (CDN).
4. The server authenticates the request, executes application code, and queries data stores.
5. The browser parses HTML and CSS, executes JavaScript, calculates layout, paints pixels, and composites layers.
6. The user interacts with the resulting interface, generating further requests and events.

The total observed delay is not necessarily the sum of every possible delay because some operations overlap. Nevertheless, a useful conceptual model is:

Here, T_user is perceived time to a specified user-visible outcome, and the other terms represent network transfer and waiting, server processing, parsing, JavaScript execution, layout, and painting. The model is approximate: browser work can overlap with network activity, and caching can eliminate some terms. It is useful for locating a dominant cost, not for exact prediction.

Two performance measures must be distinguished. Latency is the time required to complete one operation. Throughput is the amount of work completed per unit time, such as requests per second. A system can have high throughput but poor individual latency, or low latency for a small load but inadequate throughput under concurrency.

The arithmetic mean is often misleading for web latency because a small number of very slow requests can harm users. Percentiles are more informative. A p95 latency of 800 ms means that 95% of observations are at or below 800 ms and 5% are slower. Teams should inspect p50, p75, p95, and p99, segmented by geography, device, browser, route, and operation.

2. Browser Rendering and Core Web Vitals

2.1 The rendering pipeline

The browser constructs a Document Object Model (DOM) from HTML and a CSS Object Model (CSSOM) from stylesheets. It combines them into a render tree, calculates geometry, paints visual elements, and composites layers for display.

JavaScript can interrupt this pipeline. Synchronous scripts may block HTML parsing. Reading layout information immediately after changing styles can cause forced synchronous layout, sometimes called layout thrashing. Repeatedly changing and reading layout in a loop may cause many expensive recalculations.

Good practice includes:

• place critical content early in the HTML;
• defer non-critical scripts using defer or appropriate module loading;
• batch DOM reads separately from DOM writes;
• animate transform and opacity where suitable rather than layout-heavy properties such as width or top;
• avoid unnecessarily deep or large DOM trees;
• reserve dimensions for images, advertisements, and embedded content.

2.2 Core Web Vitals

Core Web Vitals are user-centred metrics used by web engineers and search platforms. Their definitions and thresholds can evolve, so teams should consult current official specifications. The principal metrics are:

LCP is affected by time to first byte, resource discovery, transfer, decoding, and rendering. INP is affected by input delay, event processing, and presentation delay. CLS is a unitless score based on layout-shift events; it is not a duration.

A metric must be interpreted with its population. Laboratory testing provides repeatable conditions and is useful for regression testing. Real User Monitoring (RUM) reflects actual devices, networks, and user behaviour. Both are necessary: laboratory tests help explain causality, while RUM reveals production experience.

2.3 Measurement workflow

A disciplined workflow is:

1. Define a user journey, such as opening a product page and adding an item to a cart.
2. Establish a baseline in a controlled environment.
3. Collect RUM data with privacy-respecting instrumentation.
4. Segment results by route, device, connection, and geography.
5. Form a hypothesis about the largest cost.
6. Change one major factor or create a controlled experiment.
7. Re-measure both the target metric and possible regressions.

Browser developer tools, Lighthouse-like laboratory tools, the Performance API, server access logs, and RUM platforms can be combined. A score without a diagnostic trace is not an explanation.

3. Network Delivery and Caching

3.1 Resource delivery

Performance is affected by request count, request size, connection reuse, protocol behaviour, and server location. HTTP/2 and HTTP/3 permit multiplexed requests, but they do not make large or unnecessary resources free. A smaller payload generally reduces transfer and parsing costs.

Compression should be negotiated using appropriate response headers. Brotli is often effective for text resources, while gzip remains widely supported. Already compressed formats such as JPEG, WebP, AVIF, and many video formats usually gain little from additional compression. Compression consumes CPU, so it should be evaluated against response size and server load.

A CDN places cacheable content closer to users and can reduce origin traffic. It is especially useful for versioned static assets, images, stylesheets, scripts, and sometimes safely cacheable API responses.

3.2 HTTP caching

Caching requires a clear policy. Important concepts include:

• Cache-Control: max-age=n: permits reuse for a specified number of seconds;
• public and private: indicate whether shared caches may store a response;
• no-store: instructs caches not to store the response;
• validators such as ETag and Last-Modified: allow revalidation;
• Vary: indicates which request headers affect the representation.

A strong production strategy is content-addressed or fingerprinted assets, such as app.8f31c.js. When the file changes, its name changes, allowing a long freshness lifetime. HTML often receives a shorter lifetime because it points to current asset names. Sensitive, personalised, or rapidly changing responses require special care; accidental shared caching can disclose private information.

Cache invalidation is difficult because copies may exist in browsers, CDNs, reverse proxies, and service workers. Versioned filenames reduce dependence on emergency purges. A service worker should have an explicit update and rollback strategy, since an outdated worker can continue serving defective assets.

4. Images, Fonts, and Bundles

Images frequently dominate page bytes. Effective practices include:

• selecting dimensions appropriate to the rendered size;
• using responsive images with srcset and sizes;
• choosing modern formats after testing visual quality and decoding cost;
• lazy-loading below-the-fold images;
• eagerly loading only genuinely critical images;
• specifying width and height or an aspect-ratio container;
• avoiding large background images when a responsive image element is more suitable.

Image optimisation is not simply “use the smallest file.” Excessive compression can harm accessibility and brand communication, while complex formats may increase decoding work on low-powered devices.

JavaScript bundles should be analysed by route and dependency. Code splitting loads only the code needed for an initial view. Tree shaking removes unused statically detectable exports, although side effects can limit its effectiveness. Dynamic imports can defer expensive features. However, excessive splitting may create request overhead and complicated loading behaviour. The objective is not the smallest individual file but a fast and reliable critical path.

A bundle budget can specify limits for initial JavaScript, CSS, image bytes, or long tasks. Budgets are guardrails, not universal laws. They should be connected to user journeys and enforced in continuous integration.

Fonts can delay text rendering or cause layout changes. Use a limited number of weights, subset large character sets, preload only critical fonts, and choose a sensible fallback strategy. font-display should be selected according to readability and visual requirements.

5. Server and Database Performance

5.1 From request to query

A slow page may be caused by application logic, connection-pool exhaustion, lock contention, inefficient serialization, or a database query. Server timing should therefore distinguish queue time, application time, database time, and external-service time.

A database index is an access structure that can reduce the records examined for a suitable predicate or ordering. Indexes are not universally beneficial: they consume storage, increase write cost, and may not help when a query returns a large proportion of a table or applies a non-selective expression.

Query plans reveal whether the database uses an index, performs a sequential scan, sorts large results, or joins inefficiently. Engineers should use EXPLAIN or an equivalent plan facility, measure with realistic data, and avoid interpreting a plan without understanding the schema and workload.

Common problems include:

• the N+1 query pattern, in which one query retrieves parent records and one additional query is issued for each parent;
• selecting unnecessary columns;
• missing or unsuitable indexes;
• unbounded result sets;
• functions on indexed columns that prevent effective index use;
• long transactions and lock contention;
• connection pools that are too large for the database capacity.

Pagination should be deliberate. Offset pagination is simple but may become expensive for large offsets and can shift under concurrent writes. Keyset pagination uses a stable ordering and a continuation key, often providing more consistent performance for large datasets.

5.2 A basic capacity relationship

A useful operational approximation is Little’s Law:

L is the average number of items in a stable system, λ is average throughput in items per unit time, and W is average time in the system. For a web service, L may represent concurrent requests, λ requests per second, and W average request duration in seconds. The units are consistent: requests = requests/second × seconds.

The relationship assumes a stable long-run system and averages over a meaningful interval. It does not by itself predict tail latency or prove that a system is healthy. For example, at 200 requests per second and an average duration of 0.25 seconds:

This suggests approximately 50 requests in service on average, not a required exact pool size.

6. Reliability Engineering

Reliability is broader than uptime. A service may be available but return incorrect prices, stale permissions, or corrupted data. Reliability objectives should include correctness, availability, latency, durability, and recoverability.

6.1 SLIs, SLOs, and error budgets

A Service Level Indicator (SLI) is a measured aspect of service behaviour, such as the proportion of valid requests completed successfully within a latency threshold. A Service Level Objective (SLO) is a target for that indicator. A Service Level Agreement (SLA) is a contractual commitment that may include consequences.

An error budget is the allowed unreliability over a period. If an SLO requires 99.9% successful availability over a 30-day month, the unavailable allowance is approximately:

This calculation assumes a 30-day month and defines failure consistently. Different SLO windows, exclusions, and measurement methods produce different allowances. Error budgets support balanced decisions: when the budget is being consumed rapidly, the organisation may prioritise reliability work over risky feature delivery.

6.2 Defensive patterns

• Timeouts: prevent indefinite waiting. Every network dependency should have a bounded timeout appropriate to the user journey.
• Retries: useful for transient failures, but dangerous when applied blindly. Use limited attempts, exponential backoff, and jitter; retry only operations that are safe or idempotent.
• Circuit breakers: temporarily stop calls to a failing dependency, allowing recovery and preventing cascading overload.
• Bulkheads: isolate resources so one tenant, route, or dependency cannot consume everything.
• Rate limiting: protects capacity and promotes fair use.
• Graceful degradation: provide cached, partial, or read-only functionality when non-critical dependencies fail.
• Idempotency: enables safe repeated submission, especially for payments and message processing.
• Health checks: distinguish liveness from readiness. A live process may exist while being unable to serve traffic.

Retries multiply load. If each of N independent layers retries up to r times, the worst-case number of downstream attempts can grow approximately as r^N, although real implementations vary and limits may apply. This is why retry policies must be designed across the entire call chain.

7. Observability

Observability is the ability to understand internal system behaviour from externally available signals. The three traditional pillars are logs, metrics, and traces; profiling and continuous user monitoring complement them.

7.1 Logging

A useful log event answers who or what acted, when, where, with which correlation identifier, and what occurred. Structured logs, commonly JSON, are easier to query than unstructured prose. A suitable event may include timestamp, service, environment, route, request ID, trace ID, outcome, duration, and error category.

Logs should be:

• precise and consistent;
• correlated across services;
• sampled or rate-limited for high-volume repetitive events;
• classified by severity;
• free of passwords, access tokens, payment data, and unnecessary personal information.

Logging every request body is rarely appropriate. Redaction must occur before data reaches central storage, not only in a dashboard.

7.2 Metrics

Metrics aggregate behaviour and support alerting. Common categories are:

• request rate;
• error rate;
• latency distributions;
• saturation of CPU, memory, connections, queues, and database pools;
• cache hit ratio;
• business measures such as completed orders.

A metric should have a clear unit, aggregation, and ownership. High-cardinality labels, such as unrestricted user IDs, can make a metrics system expensive and difficult to query. Histograms or carefully designed distributions are preferable to averages when tail latency matters.

A practical alert should indicate user impact or imminent capacity risk. Alerting on every small fluctuation creates fatigue and encourages operators to ignore genuine incidents.

7.3 Distributed tracing

A trace follows one request across services. It consists of spans representing operations such as API handling, database access, cache lookup, or an external call. Trace context propagation allows engineers to connect a browser event, gateway request, application operation, and database span.

Tracing is especially valuable for high tail latency and dependency analysis. It must be sampled thoughtfully because storing every span may be costly. Sampling should preserve important failures and unusual slow requests rather than selecting only a uniform random subset.

7.4 Error reporting and incident response

Error-reporting tools group similar exceptions, capture stack traces, identify affected releases, and provide context such as route and browser. They should avoid sensitive values and should distinguish expected user errors from programmer defects.

An incident process includes detection, triage, mitigation, communication, recovery, and learning. A blameless post-incident review should examine contributing technical and organisational conditions, identify actions with owners and deadlines, and verify that the fix reduces recurrence.

8. An Integrated Performance and Reliability Case

Illustrative case: CampusCart, a fictional university marketplace, experiences slow product pages and intermittent checkout failures during registration week.

Background and problem

RUM shows an LCP p75 of 4.1 seconds on mobile devices, INP degradation on search interactions, and a checkout success rate below its objective. Server dashboards show moderate average CPU, but p99 latency is high. Users report duplicate order attempts.

Analysis

A trace of slow product pages reveals a large hero image, a 1.8 MB JavaScript bundle, and a database query loading reviews individually for each product. The checkout trace shows calls to inventory and payment services without strict timeouts. Retries occur at both the browser and server layers. Logs contain payment-provider error messages but no consistent request or idempotency identifier.

The team avoids assuming that average CPU proves capacity is adequate. It examines p95 and p99 latency, database connection usage, queue time, and dependency saturation. Query plans confirm the review query lacks a useful composite index and follows an N+1 pattern.

Approach and solution

The team:

1. creates responsive product images and reserves their layout dimensions;
2. splits the JavaScript bundle and defers review functionality until needed;
3. batches review retrieval and adds an index aligned with the query’s filtering and ordering;
4. introduces bounded timeouts and a single coordinated retry policy for transient inventory failures;
5. adds idempotency keys to checkout requests and stores the resulting operation status;
6. instruments traces across the gateway, application, database, and payment provider;
7. defines an SLO for successful checkout and a separate latency SLO;
8. redacts payment and personal data from structured logs;
9. load-tests registration-week traffic with realistic data and failure injection;
10. deploys gradually with rollback criteria based on RUM and error-budget consumption.

Outcome

After deployment, the product page has substantially lower transferred bytes and improved mobile LCP. Database time falls because reviews are fetched in batches. Checkout no longer creates duplicate orders when users retry. The team also discovers that a third-party payment dependency is the dominant source of tail latency and negotiates a clearer timeout and fallback contract.

Learning points

The case demonstrates that performance is cross-layer. Browser optimisation alone would not solve the database and dependency problems. Reliability controls must account for user behaviour, especially repeated submissions. Observability is not an afterthought: without correlated logs and traces, the team could not connect the visible symptoms to their causes.

9. Governance, Ethics, and Contemporary Practice

Performance monitoring can collect detailed information about devices, locations, navigation, and interaction patterns. Organisations should collect only what is necessary, explain its purpose, restrict access, apply retention limits, and comply with relevant privacy and data-protection law. Session replay and precise interaction capture require particular caution.

Accessibility must not be sacrificed for speed. Removing content, reducing contrast, disabling keyboard interaction, or forcing intrusive lazy loading may improve a narrow metric while harming users. Performance budgets should include accessibility and functional correctness as release constraints.

Artificial intelligence can assist anomaly detection, log summarisation, query recommendation, and incident investigation. However, AI-generated explanations may be wrong, and telemetry can contain confidential information. Engineers must validate recommendations, control data access, and retain human responsibility for production decisions.

Contemporary systems increasingly use edge delivery, HTTP/3, server-side rendering, streaming, partial hydration, WebAssembly, and platform observability standards. Each can improve selected workloads but also introduces complexity, debugging costs, and new failure modes. Adoption should follow measured user benefit rather than novelty.

10. Best-Practice Checklist

• Measure real user journeys, not only isolated synthetic scores.
• Use percentiles and segmentation rather than averages alone.
• Set budgets for critical bytes, long tasks, latency, and error rates.
• Optimise the critical rendering path before low-impact details.
• Version static assets and define explicit cache policies.
• Inspect database plans using production-like data.
• Bound all external calls with timeouts and carefully designed retries.
• Make non-idempotent operations safe against duplicate submission.
• Correlate logs, metrics, traces, and user-visible errors.
• Protect telemetry from sensitive-data leakage.
• Test failure, overload, rollback, and recovery—not only successful paths.
• Treat reliability and accessibility as product requirements.

Chapter Summary

Web performance, reliability, and observability form a connected engineering system. Browser performance depends on the rendering pipeline, critical resources, JavaScript work, images, fonts, and layout stability. Core Web Vitals and RUM provide user-centred evidence, while laboratory tools support repeatable diagnosis. Network performance improves through appropriate compression, CDNs, HTTP caching, versioned assets, and disciplined bundle design.

Server and database performance require separation of queue, application, dependency, and query time. Indexes, query plans, batching, pagination, and connection management must be evaluated against realistic workloads. Reliability engineering uses SLIs, SLOs, error budgets, timeouts, controlled retries, circuit breakers, bulkheads, rate limits, graceful degradation, and idempotency. Observability combines structured logs, metrics, distributed traces, profiling, and error reporting while respecting privacy and security.

The central professional principle is evidence-based improvement: define a user or business outcome, measure it, form a hypothesis, change the system safely, and verify the result. Fast systems that fail unpredictably are not reliable, and reliable systems that cannot be understood are difficult to operate. Effective web engineering addresses all three qualities together.

Key Terms

Core Web Vitals: User-centred measures of loading, responsiveness, and visual stability.

Latency: Time required for an operation to complete.

Throughput: Amount of work completed per unit time.

Percentile: A value below which a specified proportion of observations falls.

LCP: Largest Contentful Paint, a loading-performance metric.

INP: Interaction to Next Paint, a responsiveness metric.

CLS: Cumulative Layout Shift, a measure of unexpected visual movement.

RUM: Real User Monitoring using production user experiences.

CDN: Content Delivery Network that serves content from distributed locations.

Cache-Control: HTTP response directive governing caching behaviour.

Code splitting: Dividing application code into independently loaded parts.

Tree shaking: Removing statically detectable unused code.

Query plan: Database strategy for executing a query.

N+1 query: One initial query followed by an additional query per returned item.

SLI: Service Level Indicator, a measured service property.

SLO: Service Level Objective, a target for an SLI.

Error budget: Permitted unreliability implied by an SLO.

Observability: Ability to infer internal system state from outputs.

Distributed trace: Connected spans describing one request across components.

Idempotency: Property that repeating an operation has the same intended effect as performing it once.

Review Questions

Short Answer Questions

1. Distinguish latency from throughput.
2. What does CLS measure?
3. Why are percentiles often more useful than average latency?
4. State two risks of indiscriminate retries.
5. What information should a structured log event normally contain?

Long Answer Questions

1. Explain the browser rendering pipeline and discuss techniques for reducing rendering and scripting cost.
2. Compare laboratory performance testing with Real User Monitoring.
3. Explain HTTP caching, asset fingerprinting, validators, and the risks of caching personalised responses.
4. Discuss how database indexes, query plans, batching, and pagination affect web application performance.
5. Explain how SLIs, SLOs, error budgets, timeouts, circuit breakers, and graceful degradation support reliability.

Analytical / Application Questions

1. A service handles 200 requests per second with an average duration of 0.25 seconds. Use Little’s Law to estimate average concurrent requests, and explain why this is not automatically a connection-pool recommendation.
2. A product page has good server response time but poor mobile LCP and INP. Propose an evidence-based investigation plan covering browser, network, bundle, and RUM data.
3. Design an observability and reliability plan for a payment workflow. Include telemetry, privacy controls, timeout and retry policies, idempotency, alerting, and incident recovery.

Multiple Choice Questions with Answers

Question 1

Which Core Web Vital measures unexpected visual movement during page use?

A. LCP

B. INP

C. CLS

D. FCP

Correct Answer: C. CLS

Question 2

At 200 requests per second and an average duration of 0.25 seconds, what is the approximate average number of requests in the system according to Little’s Law?

A. 25

B. 50

C. 200

D. 800

Correct Answer: B. 50

Question 3

Which strategy most directly supports long-term caching of static assets while avoiding stale versions?

A. Use identical filenames for every release

B. Disable all browser caching

C. Use content-fingerprinted filenames

D. Store personalised HTML in a shared cache

Correct Answer: C. Use content-fingerprinted filenames

Question 4

Why can an N+1 query pattern harm performance?

A. It prevents all database indexes from existing

B. It issues one additional query for each returned parent item

C. It compresses database results excessively

D. It always deletes duplicate records

Correct Answer: B. It issues one additional query for each returned parent item

Question 5

What is the primary purpose of a distributed trace?

A. To replace all database indexes

B. To encrypt user passwords

C. To follow one request across multiple system components

D. To increase image resolution

Correct Answer: C. To follow one request across multiple system components

Question 6

Which retry design is generally safest for transient failures?

A. Unlimited immediate retries

B. Retries with bounded attempts, backoff, and suitable idempotency

C. Retries for every validation error

D. Retries without timeouts

Correct Answer: B. Retries with bounded attempts, backoff, and suitable idempotency

Question 7

What does an error budget represent?

A. The maximum size of an application bundle

B. The permitted unreliability implied by a service objective

C. The number of database columns allowed

D. The amount of log data retained forever

Correct Answer: B. The permitted unreliability implied by a service objective

Question 8

Which practice best reduces the risk of sensitive information leakage through logs?

A. Log every request body unchanged

B. Use unrestricted user IDs as metric labels

C. Redact sensitive data before central storage

D. Disable access controls for operators

Correct Answer: C. Redact sensitive data before central storage

Question 9

Which technique commonly improves responsiveness by reducing main-thread JavaScript work?

A. Loading all features before the first view

B. Code splitting and deferring non-critical code

C. Increasing image dimensions

D. Removing all performance measurements

Correct Answer: B. Code splitting and deferring non-critical code

Question 10

Why is graceful degradation useful in a reliable web application?

A. It ensures every dependency is always available

B. It provides limited or cached functionality when non-critical dependencies fail

C. It removes the need for monitoring

D. It guarantees zero latency

Correct Answer: B. It provides limited or cached functionality when non-critical dependencies fail