Health Technology Assessment and Digital Health Evaluation

Introduction

Health care systems continuously adopt medicines, medical devices, diagnostic technologies, telehealth services, electronic health records (EHRs), artificial-intelligence tools and digital therapeutics. Adoption, however, is not equivalent to value. A technology may be clinically effective but unaffordable, economically attractive but difficult to implement, technically interoperable but unsafe to use, or highly usable but inequitable. Health technology assessment (HTA) provides a structured approach for judging whether a technology should be adopted, funded, disinvested from or restricted to particular populations.

Contemporary HTA extends beyond traditional assessment of pharmaceuticals. Digital health technologies operate within sociotechnical systems: outcomes depend on software design, connectivity, data quality, professional workflows, patient capabilities, organisational readiness, regulation and cybersecurity. Consequently, evaluation must combine clinical, economic, organisational, legal, ethical and social perspectives.

Learning Objectives

By the end of this chapter, the learner should be able to:

1. Explain the purpose, scope and principles of HTA.
2. Distinguish efficacy, effectiveness, efficiency, value and implementation feasibility.
3. Apply clinical, economic, organisational, legal, ethical and social evaluation domains.
4. Evaluate medicines, devices, diagnostics, telehealth, EHRs and digital therapeutics.
5. Assess interoperability, usability, accessibility, privacy and cybersecurity.
6. Interpret cost-effectiveness evidence and uncertainty.
7. Design a proportionate evaluation and make a defensible managerial recommendation.

1. The Concept and Scope of Health Technology Assessment

A health technology includes any intervention used to promote, prevent, diagnose, monitor or treat illness. It may be a medicine, vaccine, diagnostic test, surgical procedure, implant, clinical pathway, information system, mobile application or artificial-intelligence decision-support tool. HTA is a multidisciplinary process that systematically evaluates the properties, effects and impacts of a technology across its life cycle.

HTA is not simply a procurement exercise. Procurement asks whether an organisation can purchase a product under specified commercial conditions. HTA asks whether the technology creates sufficient health and system value, for whom, under what conditions and compared with which alternative.

The central comparator is usually current practice, sometimes called usual care or the relevant standard of care. A new technology cannot be judged in isolation. Its incremental consequences must be compared with the consequences of the alternative that would otherwise be used.

Value is multidimensional

Important dimensions of value include:

• Health gain: survival, symptom control, functional status and health-related quality of life.
• Safety: adverse events, diagnostic errors, treatment burden and unintended consequences.
• Economic value: costs relative to outcomes and opportunity cost.
• Equity: distribution of benefits and burdens across socioeconomic, geographic, age, gender, disability and ethnic groups.
• Patient experience: convenience, autonomy, dignity, trust and shared decision-making.
• System value: effects on capacity, workforce, waiting times, coordination and resilience.
• Sustainability: environmental effects, energy use, procurement and disposal.

A technology is therefore not intrinsically “valuable.” Its value is contextual and depends on population, comparator, implementation model and decision-maker priorities.

2. The HTA Evaluation Framework

A robust assessment normally follows a sequence rather than a single test.

2.1 Define the decision problem

The assessment should specify:

• target population and disease severity;
• intervention and intended use;
• comparator or comparators;
• setting and time horizon;
• outcomes that matter to patients and decision-makers;
• perspective, such as health-system, payer, societal or patient perspective;
• budget and implementation constraints.

Poorly defined decision problems produce misleading conclusions. For example, evaluating teleconsultation against no care may exaggerate benefits when the real alternative is an in-person appointment.

2.2 Examine evidence of clinical benefit and harm

Clinical evidence may include randomised controlled trials, diagnostic-accuracy studies, pragmatic trials, cohort studies, registries, systematic reviews and real-world evidence. Internal validity concerns whether the study estimates the effect reliably in its sample. External validity concerns whether that effect applies to the intended population and setting.

A distinction is essential:

• Efficacy is performance under controlled research conditions.
• Effectiveness is performance in routine practice.
• Implementation effectiveness concerns whether the intervention can be delivered consistently and reach the intended users.

For diagnostic technologies, sensitivity and specificity are useful but incomplete. Sensitivity is the probability of a positive test among people with disease; specificity is the probability of a negative test among people without disease. Predictive values depend on disease prevalence. A highly accurate test may still provide little value in a low-risk population if false positives cause anxiety, unnecessary procedures and cost.

For predictive algorithms, discrimination, calibration and clinical utility should be distinguished. Discrimination concerns separation between people with and without an outcome; calibration concerns agreement between predicted and observed risks; clinical utility concerns whether using the prediction changes decisions and outcomes beneficially.

2.3 Evaluate safety

Safety assessment should include known adverse events, foreseeable misuse, failure modes, human factors, cybersecurity threats and effects of automation. A digital system can introduce new hazards even when intended to reduce error. Examples include alert fatigue, incorrect data mapping, delayed escalation, automation bias and unsafe downtime procedures.

Risk should be assessed throughout the life cycle: design, testing, deployment, routine use, updates, maintenance, decommissioning and data deletion. Post-market surveillance is particularly important for adaptive software and artificial-intelligence systems whose performance may change when populations, workflows or data distributions change.

3. Economic Evaluation and Decision Analysis

Economic evaluation compares alternative courses of action in terms of both costs and consequences. Common forms include:

3.1 Incremental cost-effectiveness ratio

The principal comparison is the incremental cost-effectiveness ratio (ICER):

Here, C₁ and C₀ are costs of the new and comparator interventions, respectively, and E₁ and E₀ are their effects. Costs must be expressed in the same currency and price year; effects must use the same outcome unit. The ICER is meaningful only when the new intervention costs more and produces more effect. If it costs less and produces more effect, it is dominant. If it costs more and produces less effect, it is dominated.

For cost-utility analysis, the outcome may be quality-adjusted life-years (QALYs). A QALY combines length and quality of life. If a person experiences utility u for time t measured in years, the simplified QALY is:

Utility is commonly scaled from 0, representing death, to 1, representing perfect health, although some health states may be valued below zero. The formula assumes utility remains constant during the interval; changing utility requires summing or integrating utility over time.

Worked example

An electronic medication-management system costs £120,000 more than current practice over the evaluation period and is expected to prevent 30 serious medication-related admissions. The incremental cost per admission prevented is:

If the system instead generates 0.08 additional QALYs per patient for 2,000 patients, total incremental QALYs are:

The corresponding cost per QALY is £120,000 ÷ 160 = £750 per QALY. This calculation does not by itself establish adoption. Decision-makers must examine evidence quality, implementation costs, distributional effects, uncertainty and affordability.

3.2 Discounting, uncertainty and affordability

Costs and health outcomes occurring in future years may be discounted according to the relevant national guidance. Discounting reflects time preference and should not be applied casually or inconsistently. Economic models also contain parameter, structural and methodological uncertainty. Common approaches include one-way sensitivity analysis, scenario analysis and probabilistic sensitivity analysis.

A cost-effective technology may still be unaffordable if many patients become eligible simultaneously. Budget impact analysis therefore estimates expenditure over a specified horizon, incorporating uptake, eligible population, price, staffing, infrastructure and substitution effects. Cost-effectiveness addresses value for money; budget impact addresses whether the organisation can manage the financial consequences.

4. Evaluating Different Technology Classes

4.1 Medicines

Medicine assessment usually examines comparative efficacy, safety, quality of evidence, treatment adherence, dosing, interactions and long-term outcomes. Important questions include whether trial participants resemble local patients, whether surrogate endpoints predict outcomes that matter to patients and whether benefits persist beyond the trial period.

For high-cost or uncertain medicines, managed entry agreements may link reimbursement to evidence generation, clinical outcomes, expenditure caps or restricted use. Such agreements require clear data definitions, governance and exit criteria; otherwise they may merely postpone difficult decisions.

4.2 Medical devices

Devices often involve learning curves, operator dependence, maintenance, consumables and replacement cycles. Evidence from a specialist centre may not generalise to smaller hospitals. Assessment should include training requirements, calibration, servicing, physical infrastructure, usability, device failure and compatibility with existing equipment.

A device may improve technical accuracy but worsen outcomes if it increases procedure time, interrupts workflow or is difficult to use safely under pressure. Human-factors engineering is therefore central rather than optional.

4.3 Diagnostics

A diagnostic test should be assessed along the entire clinical pathway. The sequence is commonly:

Figure: Diagnostic value depends on the pathway from test accuracy to patient outcome.

Decision-curve analysis and net-benefit approaches can help determine whether a test improves decisions at clinically relevant risk thresholds. Assessment should also consider turnaround time, laboratory capacity, confirmatory testing, incidental findings and communication of uncertain results.

4.4 Telehealth

Telehealth includes synchronous video or telephone consultation, asynchronous messaging, remote monitoring and virtual wards. Evaluation should consider clinical appropriateness, continuity, escalation pathways, digital access, privacy, patient preference and the possibility of digital exclusion. A virtual service that increases access for technologically confident patients but excludes people with low connectivity may widen inequity.

Key indicators include access time, completion rate, unplanned in-person visits, emergency escalation, diagnostic concordance, patient-reported experience, clinician workload and total cost. A comparison with in-person care should account for avoided travel and time costs as well as platform, staffing and integration costs.

4.5 Electronic health records

EHR evaluation must examine clinical outcomes and information-system performance. Relevant domains include completeness, timeliness, data accuracy, medication reconciliation, decision support, documentation burden, clinician workload, downtime resilience and information exchange across organisations.

Interoperability has several layers:

• Technical interoperability: systems can connect and transmit data.
• Syntactic interoperability: data use compatible structures and formats.
• Semantic interoperability: exchanged data have the same meaning.
• Organisational interoperability: processes, responsibilities and policies support useful exchange.

Use of a common interface does not guarantee semantic interoperability. A coded field labelled “active medication” may mean a prescribed medicine in one system and a medicine actually being taken in another. Governance must therefore define data models, terminology, provenance, consent and correction procedures.

4.6 Digital therapeutics

Digital therapeutics deliver evidence-based behavioural or clinical interventions through software, often with patient interaction, feedback and monitoring. Assessment should distinguish a therapeutic intervention from a wellness application or information resource. Evidence should address engagement, adherence, clinical outcomes, durability, adverse effects, accessibility, clinical supervision and integration with professional care.

Engagement metrics, such as log-ins or screen time, are process measures rather than health outcomes. High engagement may reflect benefit, confusion or excessive burden. Evaluation should prioritise patient outcomes and examine whether the digital intervention works for groups with different literacy, language, disability and socioeconomic characteristics.

5. Usability, Interoperability and Cybersecurity

5.1 Usability and human factors

Usability is the extent to which specified users achieve specified goals with effectiveness, efficiency and satisfaction in a specified context. Evaluation should involve representative users performing realistic tasks, not merely technical demonstrations. Measures may include task completion, time, error rate, learnability, cognitive workload and satisfaction.

Human-centred design follows an iterative cycle: understand users and context, specify requirements, create prototypes, test with users, revise and evaluate in realistic conditions. Accessibility should include screen-reader compatibility, captioning, alternative input, language support, readable design and accommodation of cognitive or sensory impairment.

5.2 Cybersecurity

Cybersecurity is a patient-safety issue. A ransomware attack can delay treatment; compromised device settings can harm patients; weak identity controls can expose confidential information. A security assessment should address confidentiality, integrity and availability, often called the CIA triad.

Practical controls include least-privilege access, multifactor authentication, encryption in transit and at rest, secure configuration, patch management, network segmentation, audit logging, vulnerability disclosure, backups, incident response and tested recovery. Security must be proportionate to risk but cannot be treated as a final procurement checklist.

5.3 Privacy and data governance

Evaluation should establish the lawful basis for processing, purpose limitation, data minimisation, retention, secondary-use conditions, data-subject rights and arrangements with vendors. Health data are sensitive, and de-identification reduces but does not always eliminate re-identification risk. Data governance should specify who is responsible for quality, access, correction, breach notification and algorithm monitoring.

6. Organisational, Legal, Ethical and Social Assessment

Organisational assessment

An implementation assessment asks whether the organisation has the capability to adopt the technology. Important factors include leadership, workforce skills, workflow redesign, training, infrastructure, procurement, technical support, incentives and evaluation capacity. Frameworks such as CFIR and the Consolidated Framework for Implementation Research can structure analysis of intervention characteristics, inner setting, outer setting, individuals and implementation process.

A useful distinction is between implementation fidelity and adaptation. Fidelity protects essential mechanisms of benefit; adaptation modifies delivery to fit local context. Managers should identify which components are core and which may be safely adapted.

Legal and regulatory assessment

Legal and regulatory requirements vary by jurisdiction, but common issues include product classification, professional liability, informed consent, privacy, data retention, intellectual property, accessibility, procurement law and cross-border data transfer. Software that provides diagnosis or treatment recommendations may be regulated as a medical device or software as a medical device. The legal status should be verified with the relevant authority rather than inferred from marketing language.

Contracts should address service availability, data ownership, subcontractors, audit rights, breach response, portability, exit assistance, algorithm changes, performance commitments and deletion at contract end.

Ethical and social assessment

Ethical evaluation includes autonomy, beneficence, non-maleficence, justice, transparency and accountability. Important questions include:

• Who benefits and who bears the risks?
• Can patients refuse digital monitoring without losing essential care?
• Are automated recommendations explainable enough for responsible clinical use?
• Does the technology reproduce historical bias?
• Is data collected proportionately to the intended benefit?
• Can patients challenge or correct an automated decision?

Equity should be measured, not merely asserted. Outcomes can be stratified by relevant characteristics, while protecting privacy and avoiding inappropriate inference. Digital exclusion may arise from cost, connectivity, disability, language, trust, geography or lack of skills. A non-digital alternative may therefore be an ethical requirement rather than a temporary convenience.

7. Designing an Integrated Evaluation

An integrated evaluation combines evidence streams rather than allowing one domain to dominate automatically. A practical assessment can use the following stages:

1. Scoping: define the decision, stakeholders, population, comparator and intended use.
2. Theory of change: specify how inputs and activities are expected to produce outputs, outcomes and impacts.
3. Evidence review: assess comparative clinical evidence, economic data and implementation evidence.
4. Technical and human-factors testing: examine interoperability, usability, accessibility, reliability and security.
5. Organisational assessment: map workflows, resources, governance and training.
6. Equity and ethical review: identify differential effects and safeguards.
7. Economic modelling: estimate costs, outcomes, uncertainty and budget impact.
8. Pilot and monitoring: implement under controlled conditions with predefined success and stopping criteria.
9. Decision and reassessment: adopt, adapt, restrict, generate further evidence or reject; specify review triggers.

The evaluation plan should define outcome indicators in advance. A balanced scorecard may include clinical outcomes, patient-reported outcomes, safety events, access, equity, staff experience, financial impact, system performance and environmental effects. Indicators should have clear operational definitions, data sources, denominators, collection frequency and accountable owners.

8. Illustrative Case: Remote Monitoring for Heart Failure

Background. A regional health system considers a connected scale and symptom-monitoring application for adults recently discharged after heart-failure admission. The vendor claims that early alerts will reduce readmissions.

Problem. The technology has promising evidence from a specialist hospital, but the regional system serves older adults, rural communities and patients with limited digital literacy. It also uses an EHR supplied by a different vendor.

Analysis. The HTA team defines usual care as nurse follow-up plus access to primary care. The assessment identifies several uncertainties: whether patients will weigh themselves consistently, whether alerts will be clinically actionable, whether clinicians can respond within the proposed timeframe, and whether rural connectivity will be reliable. The economic model includes equipment, connectivity, nurse review time, integration, training, replacement and avoided admissions. It also considers false alerts and the opportunity cost of staff time.

Technical testing finds that the application can transmit data but does not consistently preserve the clinical meaning of medication changes in the EHR. Usability testing shows that patients can use the scale, but some do not understand the escalation instructions. A cybersecurity review requires multifactor authentication for staff, encrypted transmission, audit logs and a tested downtime process.

Approach. The system begins with a six-month pragmatic pilot. Eligibility is restricted to patients likely to benefit and able to consent. Non-digital follow-up remains available. Core measures include readmission rate, mortality, emergency attendance, patient-reported burden, alert response time, staff workload, equity by geography and digital access, total cost and serious incidents. A clinical governance group reviews alerts weekly and investigates any near miss.

Outcome. In this illustrative case, the pilot shows fewer readmissions among participating patients but substantial staff workload and lower participation among people with poor connectivity. The decision is not unconditional scale-up. The service is redesigned with loaned connectivity devices, telephone support, clearer instructions, alert thresholds reviewed by clinicians and an integration improvement plan. Expansion is approved only after workload and equity indicators meet predefined thresholds.

Learning points. First, promising clinical evidence does not guarantee local effectiveness. Second, interoperability includes meaning and workflow, not merely data transmission. Third, the comparator must represent actual usual care. Fourth, implementation and equity outcomes should be treated as primary decision evidence.

9. Contemporary Developments and Critical Issues

Artificial intelligence is increasingly used for image interpretation, risk prediction, documentation, triage and workflow optimisation. Evaluation should address dataset shift, subgroup performance, calibration, explainability, automation bias, human oversight and model updating. A model with high area under the receiver-operating-characteristic curve may still have poor calibration or no clinical benefit if its predictions do not change appropriate action.

Generative AI introduces additional concerns, including fabricated content, leakage of confidential data, uncertain provenance and persuasive but incorrect recommendations. Safe use requires bounded tasks, approved tools, human verification, logging, escalation and continuous monitoring.

Real-world evidence from EHRs, claims, registries and patient-generated data can complement trials, especially for long-term safety and diverse populations. However, confounding, missingness, coding changes, selection bias and measurement error limit causal interpretation. Data volume does not compensate for poor data quality or weak design.

Climate-conscious HTA is also emerging. Digital services may reduce travel but increase energy consumption, device manufacture, electronic waste and data-centre demand. Environmental consequences should be considered alongside clinical and economic value when they are material to the decision.

10. Best-Practice Principles for Managers

Managers evaluating a health technology should:

• begin with a clinical or service problem, not a product;
• define the comparator and decision perspective explicitly;
• involve patients, clinicians, technical experts, finance, legal, information governance and equity specialists;
• demand evidence appropriate to the technology’s risk and maturity;
• separate vendor claims from independently verified outcomes;
• test in the intended workflow and population;
• assess interoperability, usability, accessibility and cybersecurity before scale-up;
• establish outcome, safety and equity measures before implementation;
• plan for maintenance, updates, workforce changes and decommissioning;
• make uncertainty visible and specify conditions for reassessment.

Chapter Summary

Health Technology Assessment is a multidisciplinary process for judging the value, consequences and feasibility of health technologies. It compares an intervention with relevant alternatives and integrates clinical effectiveness, safety, economic value, organisational readiness, legal requirements, ethics, equity and social effects. Economic evaluation uses incremental comparisons, including the ICER, while budget impact analysis addresses affordability. Different technologies require tailored assessment: medicines require comparative clinical and safety evidence; devices require attention to operators and maintenance; diagnostics must be evaluated across the care pathway; telehealth requires assessment of access and escalation; EHRs require semantic and organisational interoperability; and digital therapeutics require evidence of therapeutic benefit rather than engagement alone. Usability, cybersecurity, privacy and human factors are central to patient safety. Contemporary evaluation must also address artificial intelligence, real-world evidence, digital exclusion and sustainability. A defensible adoption decision is conditional, transparent and supported by monitoring after implementation.

Key Terms

Health Technology Assessment: Systematic multidisciplinary evaluation of a technology’s effects and value for decision-making.

Efficacy: Performance under controlled research conditions.

Effectiveness: Performance in routine practice.

Comparator: The alternative against which a technology is evaluated.

QALY: Quality-adjusted life-year combining health-related quality and duration of life.

ICER: Incremental cost-effectiveness ratio comparing additional cost with additional effect.

Budget impact: Financial consequence of adopting a technology over a specified period.

Diagnostic sensitivity: Probability of a positive test among people with the condition.

Diagnostic specificity: Probability of a negative test among people without the condition.

Interoperability: Ability of systems and organisations to exchange and use information meaningfully.

Semantic interoperability: Preservation of meaning when data are exchanged.

Usability: Effectiveness, efficiency and satisfaction of specified users in a specified context.

Human factors: Study of interactions among people, technology and work systems.

Cybersecurity: Protection of information and systems from unauthorised access, alteration or disruption.

Digital therapeutics: Software-delivered interventions intended to prevent, manage or treat a condition.

Digital exclusion: Unequal ability to access or benefit from digital services.

Real-world evidence: Evidence generated from routine practice data and experiences.

Automation bias: Excessive reliance on automated recommendations.

Implementation fidelity: Delivery of essential intervention components as intended.

Review Questions

Short Answer Questions

1. What is the purpose of a comparator in HTA?
2. Distinguish efficacy from effectiveness.
3. Define semantic interoperability.
4. What does an ICER measure?
5. Why is cybersecurity a patient-safety issue?

Long Answer Questions

1. Explain how clinical, economic, organisational, legal and social domains should be integrated in an HTA.
2. Compare the evaluation requirements for telehealth, EHRs and digital therapeutics.
3. Discuss the importance of usability and human factors in medical-device assessment.
4. Explain how uncertainty and affordability influence an adoption decision.
5. Critically evaluate the ethical challenges of AI-enabled clinical decision support.

Analytical / Application Questions

1. A telehealth programme reduces travel but increases staff workload and excludes patients without broadband. Design an evaluation framework that assesses value and equity.
2. A diagnostic algorithm has high discrimination but poor calibration in older adults. What additional evidence and safeguards should be required before deployment?
3. Prepare a conditional adoption recommendation for a digital therapeutic using clinical, economic, interoperability, cybersecurity and implementation criteria.

Multiple Choice Questions with Answers

Question 1

Which statement best describes Health Technology Assessment?

A. A process concerned only with purchasing equipment

B. A multidisciplinary assessment of a technology’s effects, value and consequences

C. A method for measuring staff satisfaction only

D. A regulatory approval procedure performed only by manufacturers

Correct Answer: B. A multidisciplinary assessment of a technology’s effects, value and consequences

Question 2

An intervention costs £80,000 more than its comparator and produces 20 additional cases prevented. What is the incremental cost per case prevented?

A. £400

B. £1,600

C. £4,000

D. £16,000

Correct Answer: C. £4,000

Question 3

Which type of interoperability concerns whether exchanged data retain the same meaning?

A. Technical interoperability

B. Syntactic interoperability

C. Semantic interoperability

D. Financial interoperability

Correct Answer: C. Semantic interoperability

Question 4

Which measure combines length and quality of life?

A. Sensitivity

B. QALY

C. Budget impact

D. Specificity

Correct Answer: B. QALY

Question 5

Why should a diagnostic technology be evaluated across the care pathway?

A. Accuracy alone guarantees improved outcomes

B. The value of a test depends on interpretation, decisions and resulting patient outcomes

C. Pathway evaluation eliminates the need for clinical trials

D. Diagnostic tests do not have economic consequences

Correct Answer: B. The value of a test depends on interpretation, decisions and resulting patient outcomes

Question 6

Which is an example of automation bias?

A. A clinician independently verifies an alert

B. A patient declines digital monitoring

C. A clinician accepts an algorithmic recommendation without adequate scrutiny

D. A system records an audit trail

Correct Answer: C. A clinician accepts an algorithmic recommendation without adequate scrutiny

Question 7

Which item is primarily a budget-impact question?

A. Does the intervention improve survival compared with usual care?

B. Can the organisation finance expected expenditure during the first three years?

C. Is the test sensitive in the target population?

D. Can users complete the required task?

Correct Answer: B. Can the organisation finance expected expenditure during the first three years?

Question 8

Which cybersecurity control most directly limits access to only what a user needs?

A. Least-privilege access

B. Increasing screen brightness

C. Removing audit logs

D. Disabling backups

Correct Answer: A. Least-privilege access

Question 9

A digital therapeutic records frequent log-ins but no improvement in symptoms. What is the best interpretation?

A. Engagement automatically proves clinical effectiveness

B. Log-ins are a process measure and do not establish therapeutic benefit

C. The technology is necessarily unsafe

D. The comparator is irrelevant

Correct Answer: B. Log-ins are a process measure and do not establish therapeutic benefit

Question 10

Which design best addresses digital exclusion in a telehealth programme?

A. Require all patients to use video consultations

B. Remove telephone services to encourage adoption

C. Offer accessible alternatives and assess participation across relevant groups

D. Evaluate only patients who successfully log in

Correct Answer: C. Offer accessible alternatives and assess participation across relevant groups