Non-Functional Testing

Introduction

Functional testing asks whether a system produces the required results. Non-functional testing asks whether it produces them with acceptable qualities: quickly enough, reliably enough, securely enough, accessibly enough, and across the environments in which users actually operate. A feature may be functionally correct yet unusable because it takes 20 seconds to load, fails under concurrent demand, exposes personal data, or cannot be used with a keyboard.

Non-functional testing is therefore not a single test activity. It is a family of investigations into quality attributes, operational limits, risks, and trade-offs. Its results support engineering decisions such as capacity planning, release approval, infrastructure selection, security remediation, and service-level agreement (SLA) definition.

Learning objectives

By the end of this chapter, students should be able to:

• distinguish functional defects from non-functional limitations;
• design performance, load, stress, scalability, reliability, and resilience tests;
• interpret latency, throughput, error rate, saturation, and availability measures;
• select suitable tools for web, API, browser, security, accessibility, and compatibility testing;
• evaluate privacy and security risks ethically and legally;
• analyze test evidence and communicate release recommendations.

1. Quality attributes and testable requirements

A quality attribute becomes testable when it has a measurable context, workload, threshold, and environment. “The application must be fast” is vague. “Under 500 requests per second, 95% of product-search responses shall be below 400 ms, with an error rate below 1%” is testable.

A useful specification identifies:

1. Scenario: what operation is performed.
2. Workload: users, requests, data volume, and concurrency.
3. Environment: software, hardware, network, browser, and deployment topology.
4. Metric: what is measured.
5. Acceptance criterion: the permitted threshold or target.
6. Evidence: logs, traces, reports, screenshots, or measurements.

Non-functional requirements should be traced to business consequences. For example, a payment service may prioritize integrity and availability, while a public information site may prioritize accessibility and response time. Quality attributes can conflict: stronger encryption may increase CPU consumption; aggressive caching may improve latency but create staleness; auto-scaling may improve resilience while increasing cost.

A functional defect means that required behavior is incorrect or missing—for example, a valid coupon is rejected. An operational limitation means the behavior is correct within the tested design but unacceptable under a condition—for example, search latency exceeds the agreed threshold when traffic reaches capacity. The distinction matters because the remedies differ: a functional defect may require code correction, whereas an operational limitation may require optimization, scaling, architecture change, or a revised capacity policy.

2. Performance and capacity testing

2.1 Core concepts

Performance testing evaluates responsiveness, resource consumption, and system behavior under a defined workload. Important measures include:

• Latency: elapsed time between a request and its corresponding response. Percentiles are usually more informative than averages. The 95th percentile, p95, is the value below which 95% of observations fall.
• Throughput: completed transactions, requests, or bytes per unit of time.
• Concurrency: operations active at the same time.
• Error rate: failed operations divided by total attempted operations.
• Utilization: proportion of a resource’s capacity being consumed, such as CPU, memory, disk, database connections, or network bandwidth.
• Saturation: a condition in which a resource cannot accept additional useful work without queuing, degradation, or failure.

A basic error-rate calculation is:

This assumes that failures are clearly defined and that the numerator and denominator refer to the same observation period and operation population.

For a stable system, Little’s Law provides a useful queueing relationship:

Here, L is the average number of items in the system, λ is the average arrival or completion rate, and W is the average time an item spends in the system. Units must be consistent: requests and requests/second produce seconds. The relationship assumes a stable long-run system in which the average arrival and departure rates are equal. It does not by itself predict percentile latency or explain bottlenecks.

For example, if an API completes 120 requests per second and the average response time is 0.25 seconds, the average number of in-flight requests is:

This does not mean every moment has exactly 30 requests; it is a long-run average.

2.2 Types of performance test

Baseline testing records behavior under a small, controlled workload. It provides a comparison point for later builds.

Load testing applies an expected workload, often increasing gradually to the anticipated peak. It answers whether the system meets requirements under normal or planned demand.

Stress testing pushes beyond expected capacity to locate failure behavior and recovery characteristics. A useful stress test does not merely seek a dramatic crash; it identifies the limiting resource, error mode, and recovery path.

Spike testing introduces a sudden increase or decrease in demand. It evaluates queues, auto-scaling, connection pools, and graceful degradation.

Endurance or soak testing runs a representative workload for an extended period. It can reveal memory leaks, connection exhaustion, log growth, data corruption, and performance drift.

Volume testing examines large data sets, such as millions of records or large files.

Scalability testing studies how capacity changes when resources or instances are added. Vertical scaling adds resources to a machine; horizontal scaling adds machines or service instances. Ideal linear scaling is uncommon because coordination, shared databases, locks, network costs, and serial work constrain gains.

2.3 A disciplined performance experiment

A credible experiment has a hypothesis, controlled variables, repeatable workload, and observable results. A practical sequence is:

1. define realistic user journeys and success criteria;
2. establish a clean baseline;
3. model workload proportions, arrival patterns, data, and think time;
4. warm caches and just-in-time runtimes where appropriate;
5. ramp traffic gradually and record steady-state intervals;
6. monitor application, database, host, network, and dependency metrics;
7. repeat runs and investigate variation;
8. compare results with thresholds and prior versions;
9. identify bottlenecks rather than treating symptoms;
10. document test limitations and environmental differences.

A load generator must not become the bottleneck. Test traffic should be isolated from real users, and production-like data should be anonymized. Results from a developer laptop or a shared test server should not be represented as production capacity without qualification.

Tools include JMeter, Gatling, k6, Locust, and cloud-provider load services for HTTP/API workloads. Browser-level tools such as Lighthouse and WebPageTest assess user-facing web performance, while browser automation frameworks such as Selenium can measure journeys but are generally more expensive and variable than protocol-level load generation. Prometheus, Grafana, OpenTelemetry, and distributed tracing systems help correlate latency with infrastructure and service behavior.

2.4 Interpreting results

Averages can hide a poor user experience. Suppose 990 requests take 100 ms and 10 take 5,000 ms. The average is approximately 149 ms, but the slow tail is operationally important. Therefore reports should include p50 (median), p90, p95, p99, maximum where meaningful, throughput, errors, and resource behavior.

A performance graph should be read as a relationship, not merely as a pass/fail number. Rising latency accompanied by CPU saturation suggests a compute bottleneck. Rising latency with low CPU but exhausted database connections suggests waiting elsewhere. A sudden error increase after a queue limit is reached indicates saturation or back-pressure failure.

A capacity curve commonly has three regions: an efficient region in which throughput rises with demand; a saturation region in which latency rises sharply; and a failure region in which errors, timeouts, or rejected work dominate. The sustainable operating point should include a safety margin rather than being set exactly at the first observed failure.

3. Reliability, availability, and resilience

Reliability is the probability that a system performs required functions without failure for a specified period and environment. Availability is the proportion of time a service is usable. Resilience is the ability to withstand disruption, degrade acceptably, recover, and continue serving important functions.

A simple observed availability measure is:

The definition must specify whether planned maintenance, partial degradation, and dependency failures count as downtime. For a 30-day period with 43 minutes of unplanned downtime:

Mean time between failures (MTBF) and mean time to repair (MTTR) are operational indicators. In a simplified repairable-system model:

This assumes clearly defined failure and repair cycles and excludes factors such as logistics, planned maintenance, and complex dependencies. It should not be confused with a contractual availability figure.

Reliability testing may involve repeated execution, fault injection, long-duration operation, and recovery verification. Resilience testing deliberately interrupts components: terminate an instance, delay a dependency, revoke network access, fill disk space, or introduce malformed responses. Such work must be authorized, isolated, observable, and reversible.

Important design mechanisms include timeouts, bounded retries, exponential backoff with jitter, circuit breakers, bulkheads, idempotency keys, queues, replication, health checks, graceful degradation, backups, and tested restoration. Retrying a non-idempotent payment operation without protection can create duplicate charges; therefore resilience mechanisms must preserve business correctness as well as availability.

4. Security and privacy testing

Security testing seeks vulnerabilities that permit unauthorized access, disclosure, alteration, denial of service, or repudiation. It includes:

• threat modeling and abuse-case analysis;
• static application security testing (SAST);
• software composition analysis (SCA) for dependencies;
• dynamic application security testing (DAST);
• API authorization and authentication tests;
• configuration and infrastructure scanning;
• penetration testing by authorized personnel;
• secret, session, input-validation, and logging reviews.

Testing should address the OWASP Top 10 and API Security Top 10 as practical awareness frameworks, while recognizing that lists are not complete security standards. Tests should verify that users cannot access another user’s records by changing an identifier, that permissions are enforced server-side, that sensitive data is not exposed in logs, and that errors do not reveal unnecessary internals.

Privacy testing examines collection, purpose limitation, minimization, retention, access, correction, deletion, consent where required, and international transfer controls. Data used in test environments should be synthetic or properly de-identified. A masked name is not necessarily anonymous if it can be linked with other attributes. Privacy obligations depend on jurisdiction and context; relevant regimes may include the GDPR, the CCPA/CPRA, or sector-specific rules. Legal review is necessary because technical testing cannot determine all regulatory duties.

Security tools produce findings, not automatic proof of risk. A scanner may report a false positive, miss a business-logic flaw, or assign severity without understanding exposure. Findings should be triaged by exploitability, impact, affected assets, exposure, compensating controls, and remediation priority. Credentials, tokens, and personal data must never be placed in reports unnecessarily.

5. Accessibility testing

Accessibility testing evaluates whether people with visual, auditory, motor, cognitive, and speech-related disabilities can use the system. The Web Content Accessibility Guidelines (WCAG) provide widely used principles: perceivable, operable, understandable, and robust.

Automated tools such as axe, Lighthouse, and WAVE can identify missing labels, contrast problems, and structural issues, but automation cannot establish complete accessibility. Manual testing should include keyboard-only navigation, visible focus, logical heading structure, screen-reader interaction, zoom or reflow, captions, error recovery, and meaningful alternative text. Tests should include representative assistive technologies and, when possible, disabled users or accessibility specialists.

Accessibility is not merely a compliance exercise. It improves usability for people using mobile devices, poor connections, temporary injuries, or unfamiliar interfaces. A functionally correct form that cannot be completed without a mouse has a significant non-functional defect.

6. Compatibility and portability

Compatibility testing evaluates behavior across browsers, browser versions, operating systems, devices, screen sizes, network conditions, databases, and integrations. A test matrix should be risk-based rather than attempting every theoretical combination. Prioritize combinations according to user population, revenue, support commitments, technical differences, and historical defects.

Cross-browser testing may use Selenium WebDriver, Playwright, cloud device farms, and real devices. Responsive testing should examine layout, touch targets, orientation, zoom, slow networks, and high-density displays. API compatibility requires attention to protocol versions, character encoding, time zones, locale formats, and backward-compatible schema evolution.

Virtual devices are efficient for breadth but may not reproduce real hardware performance, sensors, keyboards, or browser behavior. Consequently, critical paths should be checked on representative physical devices.

7. Tool selection and evidence

Tool choice should follow the question, not popularity. The following mapping is useful:

A good report states build version, environment, workload, duration, data set, tool version, configuration, metrics, deviations, and raw evidence. Reproducibility is especially important when a result blocks a release.

8. Applied case: an illustrative online bookstore

Background

An online bookstore has a search API, inventory service, payment provider, and browser storefront. Functional tests pass, but customers report slow searches during marketing campaigns. The service objective is p95 search latency below 500 ms, error rate below 1%, and availability of 99.9% per calendar month.

Problem and analysis

The team first creates a representative workload: 70% searches, 20% product-page requests, and 10% sign-ins. The expected peak is 400 requests per second. A baseline at 100 requests per second produces p95 latency of 210 ms. At 400 requests per second, p95 rises to 620 ms and database connection utilization reaches 98%; at 500 requests per second, timeouts increase to 4%.

The evidence indicates a capacity limitation rather than a search correctness defect. Traces show that cache misses perform an expensive inventory query. The load generator is checked separately and has spare CPU and network capacity, reducing the likelihood that it distorted the result.

Approach and solution

The team adds an indexed query, introduces a bounded cache for stable catalog data, limits database connections per service instance, and configures a timeout and circuit breaker for the inventory dependency. It also adds a graceful response that marks stock information temporarily unavailable rather than failing the entire product page. Search results remain correct, but stock freshness is explicitly communicated.

A repeat load test at 400 requests per second produces p95 latency of 360 ms and an error rate of 0.2%. A spike test from 100 to 600 requests per second shows temporary queuing but no uncontrolled retry storm. An endurance test reveals that cache memory grows over several hours; a maximum cache size and eviction policy correct the issue. Security testing separately finds that inventory identifiers require authorization checks, and accessibility testing finds that the stock-status message is not announced by a screen reader.

Outcome and learning points

The release meets the stated performance objective under the tested workload, but the team records a capacity limit, cache freshness rule, dependency assumptions, and monitoring alerts. The case demonstrates that one test rarely proves quality. Performance, resilience, security, and accessibility interact, and a successful result requires evidence across all relevant dimensions.

9. Limitations, governance, and contemporary practice

Non-functional results are context-dependent. A test environment may differ in CPU, storage, network, data volume, traffic distribution, or dependency behavior. Synthetic workloads may omit unusual but important user journeys. Percentiles can be unstable with small samples, and averages can hide tails. A passing test is evidence about a scenario, not a universal guarantee.

Tests involving production systems require change approval, rate limits, rollback plans, monitoring, and clear ownership. Destructive resilience and penetration tests require explicit authorization. Personal data must be minimized, protected, retained only as necessary, and securely destroyed. Reports should follow least privilege and responsible disclosure practices.

Modern teams increasingly use observability-driven testing, infrastructure-as-code, ephemeral environments, continuous performance tests, and software supply-chain controls. AI-assisted tools can generate test data, summarize traces, detect anomalies, or suggest accessibility issues. However, AI outputs require human verification: generated workloads may be unrealistic, summaries may omit causal evidence, and sensitive source data may be exposed to an external service. AI should support—not replace—engineering judgment, privacy controls, and reproducible evidence.

Best-practice checklist

• Convert quality aspirations into measurable, risk-based requirements.
• Use realistic workload models and production-like but safe data.
• Measure percentiles, throughput, errors, and resource saturation together.
• Test normal load, peaks, spikes, endurance, and recovery where relevant.
• Trace symptoms to bottlenecks rather than optimizing by guesswork.
• Combine automated security and accessibility checks with expert review.
• Test compatibility using a prioritized matrix and representative real devices.
• Record environment, assumptions, limitations, and repeatability.
• Separate functional correctness from operational capacity and quality thresholds.
• Treat privacy, authorization, and test safety as release requirements.

Chapter Summary

Non-functional testing evaluates qualities that determine whether software is usable and dependable in real conditions. Performance testing measures latency, throughput, concurrency, errors, and resource behavior; load, stress, spike, endurance, volume, and scalability tests answer different questions. Little’s Law links average concurrency, throughput, and response time under stable conditions, while availability calculations require explicit definitions of downtime.

Reliability concerns failure-free operation, availability concerns usable service time, and resilience concerns resistance, graceful degradation, and recovery. Security and privacy testing must examine authorization, data exposure, dependencies, retention, and legal obligations. Accessibility requires automated and manual evaluation, and compatibility testing must use a risk-based environment matrix. Tools are valuable only when matched to a defined question and interpreted with knowledge of their limitations. The strongest practice combines measurable requirements, realistic experiments, observability, ethical governance, and evidence-based decisions.

Key Terms

Quality attribute: A measurable characteristic such as performance, security, or accessibility.

Latency: Time between an operation being initiated and its response being received.

Throughput: Completed work per unit of time.

Concurrency: Number of operations active simultaneously.

Load testing: Testing under expected or planned demand.

Stress testing: Testing beyond expected capacity to study limits and failure.

Spike testing: Testing response to sudden workload change.

Endurance testing: Long-duration testing for degradation and leaks.

Scalability: Ability to handle increased demand through added resources.

Saturation: Condition in which a resource cannot accept useful additional work.

Percentile: A value below which a specified percentage of observations fall.

Reliability: Probability of performing required functions without failure for a period.

Availability: Proportion of agreed time during which a service is usable.

Resilience: Ability to withstand disruption, degrade acceptably, and recover.

MTBF: Mean time between failures.

MTTR: Mean time to repair or restore service.

SAST: Static application security testing performed on source or compiled code.

DAST: Dynamic security testing against a running application.

WCAG: Web Content Accessibility Guidelines.

Observability: Ability to infer internal system state from outputs such as logs, metrics, and traces.

Review Questions

Short Answer Questions

1. Why can average latency be misleading in a performance report?
2. Distinguish load testing from stress testing.
3. What is the purpose of an accessibility test that uses only a keyboard?
4. State two differences between reliability and resilience.
5. Why should production data generally not be copied directly into a test environment?

Long Answer Questions

1. Explain how to design a repeatable load test for a web API, including workload, environment, measurements, and acceptance criteria.
2. Compare vertical and horizontal scalability, including technical constraints that prevent ideal linear scaling.
3. Discuss how security, privacy, and functional testing overlap but remain distinct activities.
4. Explain why automated accessibility and security tools require manual or expert review.
5. Describe how observability helps distinguish an application defect from an infrastructure or capacity limitation.

Analytical / Application Questions

1. A service completes 200 requests per second with an average response time of 0.4 seconds. Use Little’s Law to estimate average in-flight requests, and state the assumptions behind the calculation.
2. During a load test, p95 latency rises sharply while CPU remains low and database connection utilization reaches 100%. Analyze the likely bottleneck and propose two investigations.
3. An application passes functional tests but fails when a dependency is slow: retries multiply traffic and payments are duplicated. Recommend a resilience design and explain how you would test it safely.

Multiple Choice Questions with Answers

Question 1

Which measure best describes the value below which 95% of response-time observations fall?

A. Mean

B. p95

C. Maximum throughput

D. Error rate

Correct Answer: B. p95

Question 2

An API completes 80 requests per second and has an average response time of 0.5 seconds. Under Little’s Law, what is the average number of in-flight requests?

A. 40

B. 80

C. 160

D. 0.00625

Correct Answer: A. 40

Question 3

Which test is most specifically designed to reveal memory leaks over several hours?

A. Baseline test

B. Spike test

C. Endurance test

D. Compatibility test

Correct Answer: C. Endurance test

Question 4

What is the principal purpose of stress testing?

A. To verify wording in user documentation

B. To locate behavior and limits beyond expected capacity

C. To replace all functional testing

D. To test only screen-reader output

Correct Answer: B. To locate behavior and limits beyond expected capacity

Question 5

Which condition most strongly indicates resource saturation?

A. Increasing demand with stable latency and spare capacity

B. A sharp latency increase accompanied by exhausted database connections

C. A successful unit test

D. A browser with a new color theme

Correct Answer: B. A sharp latency increase accompanied by exhausted database connections

Question 6

Which statement best distinguishes resilience from availability?

A. Resilience concerns response time only, while availability concerns security only

B. Resilience concerns withstanding and recovering from disruption, while availability concerns usable service time

C. They are identical terms in all contexts

D. Availability applies only to desktop applications

Correct Answer: B. Resilience concerns withstanding and recovering from disruption, while availability concerns usable service time

Question 7

Which activity is a privacy-preserving practice for test data?

A. Publishing production credentials in a test report

B. Copying all customer records to every developer laptop

C. Using synthetic or properly de-identified data

D. Disabling authorization checks in production

Correct Answer: C. Using synthetic or properly de-identified data

Question 8

Why is manual accessibility testing still necessary when axe or Lighthouse is used?

A. Automated tools cannot detect any HTML issue

B. Manual testing evaluates interactions such as keyboard and screen-reader use that automation may miss

C. Manual testing removes the need for WCAG

D. Automated tools measure database throughput instead

Correct Answer: B. Manual testing evaluates interactions such as keyboard and screen-reader use that automation may miss

Question 9

Which tool category primarily examines third-party libraries for known vulnerabilities?

A. Software composition analysis

B. Browser rendering test

C. Endurance test

D. Screen-reader test

Correct Answer: A. Software composition analysis

Question 10

A service has 43 minutes of unplanned downtime in a 30-day month of 43,200 minutes. Approximately what availability was observed?

A. 90.05%

B. 99.005%

C. 99.9005%

D. 100.43%

Correct Answer: C. 99.9005%