End-to-End, Regression, and Exploratory Testing

Introduction

Modern web applications are distributed systems. A user action in a browser may invoke a JavaScript interface, an application programming interface (API), authentication services, databases, message queues, payment providers, analytics systems, and cloud infrastructure. Testing such an application therefore requires more than checking whether individual functions return expected values. It requires a balanced strategy that tests behaviour at several levels while controlling cost, execution time, maintenance effort, and diagnostic difficulty.

This chapter examines end-to-end, regression, and exploratory testing within that broader strategy. It explains the test pyramid and related models, distinguishes major test types, and develops a practical approach to smoke, sanity, system, acceptance, usability, compatibility, accessibility, and responsive testing. Particular attention is given to avoiding excessive dependence on end-to-end tests. Although end-to-end tests are valuable, they are comparatively slow, fragile, expensive to maintain, and often poor at identifying the precise source of failure.

The central principle is that quality is produced by a portfolio of complementary tests. Unit and component tests provide fast feedback and precise diagnosis; integration and contract tests examine boundaries; a smaller number of carefully selected end-to-end tests validate critical user journeys; and exploratory and specialist testing investigate risks that scripted automation may miss.

Learning Objectives

By the end of this chapter, you should be able to:

1. Explain the purpose and limitations of the test pyramid and related testing models.
2. Distinguish smoke, sanity, regression, system, acceptance, exploratory, usability, compatibility, accessibility, and responsive testing.
3. Construct a risk-based test suite without excessive end-to-end dependence.
4. Design stable end-to-end tests for critical user journeys.
5. Conduct exploratory testing using explicit missions, heuristics, and evidence.
6. Evaluate test results, failures, coverage, flakiness, and maintenance cost.
7. Apply testing practices responsibly, including accessibility, privacy, security, and ethical considerations.

1. Testing as a Layered Quality Strategy

A test suite is not simply a collection of test cases. It is an engineered feedback system. Each test should provide useful evidence about a product risk at an appropriate cost.

Important dimensions include:

• Scope: the amount of software and infrastructure exercised.
• Speed: how quickly feedback is obtained.
• Isolation: whether the test depends on external components.
• Diagnostic value: how easily a failure identifies its cause.
• Realism: how closely the test represents actual use.
• Maintenance cost: the effort required to keep the test valid.
• Reliability: the likelihood that the test produces the same result when the product has not changed.

No single test level optimizes all these dimensions. A unit test is fast and diagnostic but may not reveal a deployment or integration defect. An end-to-end test is realistic but may fail because of a browser, network, data, service, or timing issue. A professional strategy therefore distributes tests according to risk and feedback needs.

1.1 The test pyramid

The test pyramid, commonly associated with Mike Cohn, recommends many fast, focused tests at the base, fewer broader tests in the middle, and a small number of slow, realistic tests at the top.

The traditional layers are:

The pyramid is a heuristic, not a law. A front-end application may use many component tests and API tests rather than thousands of traditional unit tests. A data-intensive system may require substantial integration testing. The important idea is to avoid making the broadest and most expensive tests carry responsibilities better handled by narrower tests.

A useful design rule is: test a behaviour at the lowest level that can provide convincing evidence. For example, validation that an email address has a valid format belongs primarily in unit or component tests. A small number of end-to-end tests should confirm that a user can submit a valid registration form and reach the expected account page.

1.2 Related models

The testing trophy emphasizes static analysis, unit tests, integration tests, and a smaller quantity of end-to-end tests, often reflecting modern web development where integration tests provide high value. The testing honeycomb similarly emphasizes service and integration tests in distributed systems. These models do not invalidate the pyramid; they adapt its emphasis to architecture and technology.

The correct model depends on risks. For a monolithic calculator, unit tests may dominate. For a web application whose most serious failures occur between the browser, API, database, and identity provider, integration and contract tests deserve substantial investment. For any model, excessive duplication is undesirable: testing the same rule at five layers increases maintenance without necessarily increasing confidence.

2. Test Levels and the Role of End-to-End Testing

Unit and component tests

Unit tests verify a small unit of behaviour in isolation, often using controlled substitutes for external dependencies. Component tests exercise a UI component or service with selected collaborators. They are appropriate for calculations, validation rules, state transitions, rendering conditions, and error handling.

Integration and contract tests

Integration tests verify that components communicate correctly. They may use a real database, a test message broker, or an HTTP boundary. Contract tests verify agreed request and response structures between consumers and providers. These tests can detect incompatible API changes without requiring a complete browser journey.

System and end-to-end tests

A system test evaluates a complete application or a substantial subsystem against specified requirements. An end-to-end (E2E) test follows a realistic path through the application, often using a browser and deployed services. For example, an E2E test might create an account, sign in, add an item to a basket, complete a test payment, and verify an order confirmation.

E2E tests are especially valuable for:

• critical revenue or safety journeys;
• authentication and authorization flows;
• routing, deployment, and configuration failures;
• integration with essential external services;
• confirmation that independently tested layers work together.

They are poor substitutes for unit or integration tests because a failure may originate in any layer. A well-designed E2E suite is deliberately small and representative. It should not repeat every validation rule or every permutation of a feature.

Acceptance testing

Acceptance testing determines whether a product or increment satisfies stakeholder, business, contractual, or regulatory acceptance criteria. It may be performed by users, product owners, testers, or automated systems. Acceptance tests should express meaningful outcomes, such as “a customer can download an invoice,” rather than implementation details such as a particular CSS class being present.

3. Smoke, Sanity, and Regression Testing

Smoke testing

A smoke test is a shallow, broad check that determines whether a build or environment is sufficiently functional for further testing. Typical checks include application startup, basic navigation, authentication, a health endpoint, and one critical transaction. A failed smoke test usually blocks deeper testing because the environment is not trustworthy.

Sanity testing

A sanity test is a focused check of a specific change or repair. After a defect in password reset is fixed, sanity testing might verify password-reset requests, token use, expiry, and the affected confirmation page. It is narrower than a full regression suite and is intended to establish that the changed area behaves plausibly.

The terms are not universally standardized. Teams should document their local definitions rather than rely on labels alone.

Regression testing

Regression testing checks whether existing behaviour has been unintentionally damaged by a change. Regression testing can be manual or automated and can occur at every test level. A unit regression test may protect a corrected calculation; an API regression test may protect response compatibility; an E2E regression test may protect a critical business journey.

A regression suite should be curated. Tests should be added when defects reveal a durable risk, but obsolete, duplicated, or persistently unreliable tests should be repaired or removed. A large suite is not automatically a good suite.

A practical selection process is:

1. Identify changed components and their dependencies.
2. Identify user journeys and business rules affected.
3. Run fast unit, component, contract, and API tests first.
4. Run targeted regression tests around the change.
5. Run critical-path E2E tests and broader suites according to risk.
6. Review failures for product defects, test defects, environment failures, and data problems.

4. Building a Balanced Test Suite

Test design should begin with risks rather than with a target number of tests. A risk can be expressed qualitatively as a combination of impact and likelihood. Teams may use a simple risk matrix rather than a numerical formula. High-impact, high-likelihood risks receive stronger and earlier evidence.

A traceability table can connect requirements, risks, test levels, and evidence:

4.1 Managing the end-to-end budget

An organization should decide which journeys truly require full-stack evidence. Selection criteria include business criticality, technical integration complexity, defect history, regulatory significance, and user frequency. A critical journey should have a small number of independent E2E tests covering its major success and failure paths. Detailed combinations should be tested below the E2E layer.

For example, an online store might use:

• unit tests for tax, discount, and shipping calculations;
• component tests for basket controls and validation messages;
• API and contract tests for order and payment interfaces;
• a few E2E tests for guest checkout, signed-in checkout, and payment failure;
• exploratory sessions for unusual browser, network, and recovery conditions.

This arrangement is more maintainable than automating every possible product, discount, address, payment, and browser combination through the UI.

4.2 Test data and environment design

Reliable broad testing depends on deliberate test data. Data should be isolated where parallel execution occurs, reset or seeded deterministically, and identifiable as synthetic. Tests should avoid sharing mutable accounts unless sharing is itself under test. External services should use sandbox environments, simulators, or contract verification where appropriate.

Environment configuration should be observable. A test report should identify application version, browser version, operating system, feature flags, service endpoints, and test-data version. Without this information, reproducing a failure becomes unnecessarily difficult.

5. Designing Reliable End-to-End Tests

A reliable E2E test has a clear purpose, stable setup, observable outcomes, and controlled dependencies.

Arrange, act, and assert

The familiar structure is:

• Arrange: establish data, authentication state, and environment conditions.
• Act: perform a user-relevant action.
• Assert: verify a meaningful outcome, preferably at more than one relevant boundary.

Assertions should focus on observable behaviour rather than implementation details. “Order confirmation displays order number” is more stable than “element has class confirmation-panel-v2.” Selectors should use accessible roles, labels, stable test identifiers, or other intentional interfaces rather than fragile positional or styling selectors.

Synchronization and flakiness

A flaky test sometimes passes and sometimes fails without a relevant product change. Common causes include arbitrary sleeps, asynchronous requests, animation, shared data, clock dependence, network instability, and incomplete cleanup. Prefer waiting for a meaningful condition, such as a response, state transition, or visible accessible element, rather than waiting a fixed duration.

Retries can reduce noise but can also hide defects. A retry policy should record the initial failure and be used for diagnosis, not as a substitute for fixing nondeterminism. Teams should monitor flaky tests and quarantine them only temporarily, with ownership and a removal deadline.

Failure diagnosis

E2E failures should collect screenshots, browser logs, network traces, video where useful, server logs, and correlation identifiers. The objective is not merely to report red or green status; it is to reduce the time from failure to explanation.

6. Exploratory Testing

Exploratory testing is a simultaneous process of learning about the product, designing tests, executing them, and interpreting results. It is not unscripted random clicking. A disciplined exploratory session has a mission, a timebox, a product area, relevant risks, notes, and a debrief.

A session might investigate: “Explore recovery from interrupted checkout on mobile networks for 60 minutes.” The tester may vary input values, navigation order, network conditions, browser history, refreshes, permissions, and timing. The tester records actions, observations, data, environment, and suspected defects.

Useful heuristics include:

• CRUD: create, read, update, and delete operations;
• boundary analysis: empty, minimum, maximum, and unusually large values;
• SFDPOT: structure, function, data, platform, operations, and time;
• error guessing: applying knowledge of previous defects and implementation risks;
• user personas: considering novice, expert, assistive-technology, and hostile users;
• state transitions: examining valid, invalid, repeated, and interrupted transitions.

Exploratory testing complements automation. Automated tests are excellent at repeatability; human exploration is often better at noticing confusing workflows, surprising interactions, visual hierarchy problems, and combinations not anticipated by the original specification. Findings should be converted into automated regression tests when they represent repeatable, important risks.

7. Specialist System Testing

Usability testing

Usability testing examines whether representative users can accomplish tasks effectively, efficiently, and satisfactorily. Measures may include completion, errors, assistance required, time on task, and qualitative comments. A usability problem is not necessarily a software defect in the narrow technical sense, but it can prevent successful use and therefore deserves product attention.

Moderated sessions allow probing and clarification; unmoderated sessions can reach more participants but provide less contextual information. Ethical practice requires informed consent, minimization of personal data, and care not to expose participants to real financial or sensitive operations.

Compatibility testing

Compatibility testing evaluates behaviour across relevant browsers, browser versions, operating systems, devices, screen sizes, input methods, and network conditions. The matrix should be risk-based rather than exhaustive. Usage analytics, organizational support commitments, customer contracts, and known platform differences can guide selection.

Virtual devices and cloud device farms increase breadth but may not reproduce every hardware, font, camera, performance, or input characteristic. High-risk combinations should be verified on representative physical devices when feasible.

Accessibility testing

Accessibility testing evaluates whether people with disabilities can perceive, operate, understand, and navigate the application. It should align with applicable standards and law, such as the Web Content Accessibility Guidelines (WCAG) and relevant national requirements. Automated tools can identify issues such as missing labels, poor contrast, or invalid structure, but they cannot establish full accessibility.

A practical accessibility strategy combines:

• semantic HTML and correct heading structure;
• keyboard-only navigation;
• visible and logical focus order;
• screen-reader testing on representative flows;
• sufficient text and control contrast;
• meaningful alternative text;
• accessible names, roles, and states;
• zoom and reflow testing;
• checks that errors are announced and explained.

Accessibility is not an optional final inspection. It should influence design, component implementation, acceptance criteria, and exploratory missions.

Responsive testing

Responsive testing evaluates how layout and interaction adapt to viewport size, orientation, input method, and content length. Testers should examine narrow and wide screens, intermediate widths, zoom, long labels, localization, dynamic text, virtual keyboards, and touch targets. A page that looks correct at two predefined widths may still fail at an intermediate width.

The aim is not to prove every pixel identical across devices. The aim is to verify readable content, usable controls, preserved task flow, and acceptable visual hierarchy across supported conditions.

8. A Continuous Testing Workflow

A mature pipeline stages feedback according to speed and cost:

1. Pre-commit: formatting, linting, static analysis, and focused unit tests.
2. Pull request: unit, component, integration, contract, and selected API tests.
3. Build or deployment: smoke tests against a controlled environment.
4. Release candidate: targeted regression, compatibility, accessibility, and critical E2E tests.
5. Post-deployment: health checks, synthetic monitoring, and carefully selected production-safe journeys.
6. Scheduled or risk-triggered: broader regression, exploratory, usability, and device testing.

The exact order varies, but fast tests should generally provide feedback before slow tests. A failure in a lower layer should not be obscured by launching hundreds of dependent browser tests.

Useful indicators include pass rate, failure clustering, execution time, defect escape rate, test flakiness, mean time to diagnose, and coverage of important risks. Code coverage is informative but incomplete: executing a line does not prove that the assertion is meaningful or that the behaviour is correct.

9. Illustrative Case: Online Course Registration

The following is a fictional illustrative case.

Background

A university deploys a web portal through which students search for modules, enrol, pay fees where applicable, and receive confirmation. The system includes a browser front end, registration API, student database, identity provider, payment service, and email service.

Problem

After a release, some students report that enrolment appears successful but the confirmation email is missing. Others cannot complete registration using a keyboard on a narrow laptop screen. The team proposes adding more browser E2E tests for every course and payment combination.

Analysis

The proposed approach would increase execution time without directly testing the likely causes. The email problem involves an asynchronous service boundary; the keyboard problem involves semantics, focus management, and responsive layout. The team maps risks to suitable test levels.

Approach and solution

The team creates deterministic test courses and student accounts. Unit tests cover prerequisites, capacity, duplicate enrolment, and fee calculations. Integration tests verify that a successful registration creates the correct record and publishes one email event. Contract tests ensure that the email service accepts the event schema. Three E2E tests cover successful enrolment, rejection for a full course, and recovery after a payment interruption.

An exploratory session investigates keyboard navigation at narrow and intermediate widths. It discovers that focus moves behind a modal after an error and that the submit button is inaccessible when browser zoom is increased. The defects are corrected, and a component-level regression test is added for focus restoration. A smaller E2E test confirms that the complete corrected journey remains functional.

Outcome

The revised suite runs faster, identifies failures more precisely, and covers more relevant risks than the original proposal. The missing email defect can be localized to event publication or consumption without treating the entire browser journey as the diagnostic instrument. Accessibility and responsive defects are detected before release rather than being assumed to be covered by a successful desktop E2E test.

Learning points

1. A user journey can require evidence at multiple levels.
2. E2E tests should confirm integration, not duplicate every lower-level rule.
3. Accessibility and responsive quality require intentional testing.
4. Exploratory findings can be converted into durable automated regression protection.
5. Test data and service contracts are central to reliable system testing.

10. Challenges, Governance, and Contemporary Practice

Limitations and trade-offs

Testing cannot prove the complete absence of defects. A passing suite may reflect incomplete requirements, untested environments, unrealistic data, or assertions that are too weak. Excessive automation can create maintenance debt, while insufficient automation can make regression unsafe. Manual testing is not automatically exploratory, and automation is not automatically objective or unbiased.

Security, privacy, and governance

Test environments must not expose production personal data unnecessarily. Data should be masked, minimized, access-controlled, and retained only as long as needed. Test credentials and service tokens belong in secure secret-management systems, not source repositories. Payment testing should use provider-approved sandbox mechanisms. Screenshots, videos, and logs may contain personal information and require appropriate controls.

Accessibility results should be treated as quality and compliance evidence, but automated conformance reports should not be presented as proof of complete accessibility. Defect severity should consider harm, affected populations, legal obligations, and barriers to essential services.

AI-assisted testing

AI-based tools can help generate test ideas, summarize failures, identify unusual patterns, or produce draft test data. They can also generate redundant tests, misunderstand requirements, reproduce insecure data, or create false confidence. Human review remains necessary. Teams should validate generated tests against acceptance criteria, protect confidential source and user data, and track whether generated assertions actually detect meaningful failures.

Best practices

• Begin with risks, requirements, and user outcomes.
• Keep most checks at the fastest level that provides credible evidence.
• Use E2E tests for a small set of critical, representative journeys.
• Prefer stable conditions and meaningful synchronization over arbitrary delays.
• Make test data deterministic, isolated, and privacy-safe.
• Treat flaky tests as defects in the delivery system.
• Combine automation with exploratory, usability, accessibility, compatibility, and responsive testing.
• Record environment information and diagnostic evidence.
• Review suites periodically for duplication, obsolescence, and missing risks.
• Convert important exploratory discoveries into appropriate regression tests.

Chapter Summary

Effective testing of full-stack applications requires a layered portfolio rather than dependence on browser journeys. The test pyramid and related models guide teams toward many fast unit, component, integration, and contract tests, supported by fewer but valuable system and end-to-end tests. E2E testing is most appropriate for critical user-visible workflows, deployment confidence, and interactions that cannot be credibly verified at a narrower level.

Smoke testing determines whether an environment is testable; sanity testing focuses on a recent change; regression testing protects existing behaviour. System and acceptance testing assess complete outcomes, while exploratory testing uses structured investigation to discover unexpected risks. Usability, compatibility, accessibility, and responsive testing address quality dimensions that ordinary functional scripts may overlook.

A balanced suite is risk-based, diagnostically useful, reliable, privacy-conscious, and continuously maintained. Its success is measured not by test count alone but by the quality and speed of evidence it provides about important user and business risks.

Key Terms

Test pyramid: A model recommending many fast, focused tests and fewer broad, expensive tests.

End-to-end testing: Testing a complete user-relevant workflow across the integrated application.

Regression testing: Testing existing behaviour after change to detect unintended damage.

Smoke testing: A shallow check that determines whether a build or environment is suitable for deeper testing.

Sanity testing: Focused testing of a changed or repaired area.

Exploratory testing: Simultaneous learning, test design, execution, and interpretation.

System testing: Evaluation of a complete application or substantial subsystem.

Acceptance testing: Testing whether specified stakeholder or business outcomes are satisfied.

Component testing: Testing a UI or service component with controlled collaborators.

Contract testing: Verification that communicating services obey an agreed interface.

Flaky test: A test whose result varies without a relevant product change.

Test oracle: A basis for deciding whether an observed result is correct.

Test data: Inputs and environmental records used to execute tests.

Usability testing: Evaluation of how effectively and satisfactorily users accomplish tasks.

Compatibility testing: Evaluation across browsers, devices, operating systems, and conditions.

Accessibility testing: Evaluation of use by people with diverse abilities and assistive technologies.

Responsive testing: Evaluation of adaptation to viewport, orientation, content, and input conditions.

Risk-based testing: Prioritizing test effort according to potential impact and likelihood.

Test isolation: Preventing one test's state or data from improperly affecting another.

Test flakiness: Unreliability caused by nondeterministic tests or environments.

Review Questions

Short Answer Questions

1. What is the principal purpose of the test pyramid?
2. Distinguish smoke testing from sanity testing.
3. Why are E2E tests generally more difficult to diagnose than unit tests?
4. What is meant by a flaky test?
5. Name two activities that should form part of accessibility testing.

Long Answer Questions

1. Explain how a team can construct a balanced test suite for a full-stack web application without excessive E2E dependence.
2. Compare unit, integration, contract, system, and acceptance testing in terms of scope, feedback, and limitations.
3. Describe a disciplined exploratory testing session, including mission, heuristics, evidence, and follow-up.
4. Explain how smoke, sanity, and regression testing may be arranged in a continuous delivery pipeline.
5. Discuss the technical, ethical, and governance issues involved in test data and automated browser testing.

Analytical / Application Questions

1. A checkout suite contains 400 browser tests and takes four hours, while pricing defects are frequently found only after release. Propose a redistribution of tests across levels and justify your decisions.
2. A registration E2E test passes on desktop Chrome but fails intermittently on a mobile device after a slow network response. Analyze possible causes and design an investigation plan.
3. Design a risk-based test strategy for a responsive banking transfer feature that includes accessibility, compatibility, exploratory, regression, and acceptance testing.

Multiple Choice Questions with Answers

Question 1

Which test level is generally most appropriate for verifying a discount calculation in isolation?

A. Unit testing

B. End-to-end testing

C. Usability testing

D. Compatibility testing

Correct Answer: A. Unit testing

Question 2

What is the primary purpose of smoke testing?

A. To test every business rule in depth

B. To determine whether a build or environment is suitable for further testing

C. To measure user satisfaction

D. To verify all supported browsers

Correct Answer: B. To determine whether a build or environment is suitable for further testing

Question 3

Which practice best reduces end-to-end test flakiness?

A. Using arbitrary long sleeps

B. Sharing mutable accounts across parallel tests

C. Waiting for meaningful application conditions

D. Removing assertions from unstable tests

Correct Answer: C. Waiting for meaningful application conditions

Question 4

What does contract testing primarily verify?

A. That users find an interface attractive

B. That communicating services follow an agreed interface

C. That a page renders at every screen width

D. That a database has no duplicate records

Correct Answer: B. That communicating services follow an agreed interface

Question 5

Which statement best describes exploratory testing?

A. Random clicking without recording results

B. Testing only after all automated tests have passed

C. Simultaneous learning, test design, execution, and interpretation

D. Repeating a fixed script without variation

Correct Answer: C. Simultaneous learning, test design, execution, and interpretation

Question 6

Which is an appropriate accessibility testing activity?

A. Checking only whether the page loads in Chrome

B. Testing whether the complete flow can be operated by keyboard

C. Measuring server CPU usage

D. Deleting all error messages

Correct Answer: B. Testing whether the complete flow can be operated by keyboard

Question 7

A sanity test is most likely to be performed after which event?

A. A specific defect has been fixed

B. A new programming language has been invented

C. A usability study has recruited participants

D. A browser is removed from the support matrix

Correct Answer: A. A specific defect has been fixed

Question 8

Why should a regression suite be curated periodically?

A. To increase duplication deliberately

B. To remove all automated tests

C. To eliminate obsolete, duplicated, or unreliable tests

D. To ensure every test is end-to-end

Correct Answer: C. To eliminate obsolete, duplicated, or unreliable tests

Question 9

Which assertion is generally more stable in an end-to-end test?

A. A button has a particular CSS class name

B. The confirmation page displays an order number

C. The fourth div has a specific pixel position

D. A page uses a particular internal component name

Correct Answer: B. The confirmation page displays an order number

Question 10

What is the main reason to use risk-based selection for compatibility testing?

A. It makes unsupported platforms appear supported

B. It avoids testing any physical devices

C. It focuses effort on combinations most important to users and consequences

D. It guarantees that every possible device is tested

Correct Answer: C. It focuses effort on combinations most important to users and consequences