Value of Information Analytical Methods: Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force

Rapid Assessment of the Need for Evidence: Applying the Principles of Value of Information to Research Prioritisation

We propose a short-cut heuristic approach to rapidly estimate value of information (VOI) using information commonly reported in a research funding application to make a case for the need for further evaluative research. We develop a "Rapid VOI" approach, which focuses on uncertainty in the primary outcome of clinical effectiveness and uses this to explore the health consequences of decision uncertainty. We develop a freely accessible online tool, Rapid Assessment of the Need for Evidence (RANE), to allow for the efficient computation of the value of research. As a case study, the method was applied to a proposal for research on shoulder pain rehabilitation. The analysis was included as part of a successful application for research funding to the UK National Institute for Health and Care Research. Our approach enables research funders and applicants to rapidly estimate the value of proposed research. Rapid VOI relies on information that is readily available and reported in research funding applications. Rapid VOI supports research prioritisation and commissioning decisions where there is insufficient time and resources available to develop and validate complex decision-analytic models. The method provides a practical means for implementing VOI in practice, thus providing a starting point for deliberation and contributing to the transparency and accountability of research prioritisation decisions.

It Takes 2 to Tango. Setting Out the Conditions in Which Performance-Based Risk-Sharing Arrangements Work for Both Parties

OBJECTIVES

Faster regulatory approval processes often fail to achieve faster patient access. We seek an approach, using performance-based risk-sharing arrangements, to address uncertainty for payers regarding the relative effectiveness and value for money of products launched through accelerated approval schemes. One important reason for risk sharing is to resolve differences of opinion between innovators and payers about a technology's underlying value. To date, there has been no formal attempt to set out the circumstances in which risk sharing can address these differences.

METHODS

We use a value of information framework to understand what a performance-based risk-sharing arrangements can, in principle, add to a reimbursement scheme, separating payer perspectives on cost-effectiveness and the value of research from those of the innovator. We find 16 scenarios, developing 5 rules to analyze these 16 scenarios, identifying cases in which risk sharing adds value for both parties.

RESULTS

We find that risk sharing provides an improved solution in 9 out of 16 combinations of payer and innovator expectations about treatment outcome and the value of further research. Among our assumptions, who pays for research and scheme administration costs are key.

CONCLUSIONS

Steps should be undertaken to make risk sharing more practical, ensuring that payers consider it an option. This requires additional costs to the health system falling on the innovator in an efficient way that aligns incentives for product development for global markets. Health systems benefits are earlier patient access to cost-effective treatments and payers with higher confidence of not wasting money. Innovators get greater returns while conducting research.

Net Monetary Benefit Lines Augmented with Value-of-Information Measures to Present the Results of Economic Evaluations under Uncertainty

BACKGROUND

Methods to present the result of cost-effectiveness analyses under parameter uncertainty include cost-effectiveness planes (CEPs), cost-effectiveness acceptability curves/frontier (CEACs/CEAF), expected loss curves (ELCs), and net monetary benefit (NMB) lines. We describe how NMB lines can be augmented to present NMB values that could be achieved by reducing or resolving parameter uncertainty. We evaluated the ability of these methods to correctly 1) identify the alternative with the highest expected NMB and 2) communicate the magnitude of parameter and decision uncertainty.

METHODS

We considered 4 hypothetical decision problems representing scenarios with high variance or correlated cost and effect estimates and alternatives with similar cost-effectiveness ratios. We used these decision problems to demonstrate the limitations of existing methods and the potential of augmented NMB lines to resolve these issues.

RESULTS

CEPs and CEACs/CEAF could falsely imply the lack of sufficient evidence to identify the optimal option if cost and effect estimates have high variance, are correlated across alternatives, or when alternatives have similar cost-effectiveness ratios. The augmented NMB lines and ELCs can correctly identify the option with the highest expected NMB and communicate the potential benefit of resolving uncertainties. Like ELCs, the augmented NMB lines provide information about the value of resolving parameter uncertainties, but augmented NMB lines may be easier to interpret for decision makers.

CONCLUSIONS

Our analysis supports recommending the augment NMB lines as an important method to present the results of economic evaluation studies under parameter uncertainty.

HIGHLIGHTS

The results of cost-effectiveness analyses (CEAs) when the cost and effect estimates of alternatives are uncertain are commonly presented using cost-effectiveness planes (CEPs), cost-effectiveness acceptability curves/frontier (CEACs/CEAF), and expected loss curves (ELCs).Although currently not often used, net monetary benefit (NMB) lines could present the results of cost-effectiveness to identify the alternative with the highest expected NMB values given the current level of uncertainty. Furthermore, NMB lines can be augmented to 1) show metrics of value of information, which measure the value of additional research to reduce or eliminate the decision uncertainty, and 2) display the confidence intervals along the NMB lines to ensure that NMB values are estimated accurately using a sufficiently large number of parameter samples.Using several decision problems, we demonstrate the limitation of existing methods to present the results of CEAs under parameter uncertainty and how augmented NMB lines could resolve these issues.Our analysis supports recommending augmented NMB lines as an important method to present the results of CEA under uncertainty since they 1) correctly identify the alternative with the highest expected NMB value given the current evidence, 2) provide information about the potential value of additional research to improve the decision by reducing or resolving uncertainty in model parameters, 3) assist the analysis to visually ensure that enough parameter samples are used to estimate the expected NMB of alternatives, and 4) are easier to interpret for decision makers compared with other methods.

A Tutorial on Value-Based Adaptive Designs: Could a Value-Based Sequential 2-Arm Design Have Created More Health Economic Value for the Big CACTUS Trial?

OBJECTIVES

Value-based trials aim to maximize the expected net benefit by balancing technology adoption decisions and clinical trial costs. Adaptive trials offer additional efficiency. This article provides guidance on determining whether a value-based sequential design is the best option for an adaptive 2-arm trial, illustrated through a case study.

METHODS

We outlined 4 steps for the value-based sequential approach. The case study re-evaluates the Big CACTUS trial design using pilot trial data and a model-based health economic analysis. Expected net benefit is computed for (1) original fixed design, (2) value-based design with fixed sample size, and (3) optimal value-based sequential design with adaptive stopping. We compare pretrial modeling with the actual Big CACTUS trial results.

RESULTS

Over 10 years, the adoption decision would affect approximately 215 378 patients. Pretrial modeling shows that the expected net benefit minus costs are (1) £102 million for the original fixed design, (2) £107 million (+5.3% higher) for the value-based design with optimal fixed sample size, and (3) £109 million (+6.7% higher) for the optimal value-based sequential design with maximum sample size of 435 per arm. Post hoc analysis using actual Big CACTUS trial data indicates that the value-adaptive trial with a maximum sample size of 95 participant pairs would not have stopped early. Bootstrap simulations reveal a 9.76% probability of early completion with n = 95 pairs compared with 31.50% with n = 435 pairs.

CONCLUSIONS

The 4-step approach to value-based sequential 2-arm design with adaptive stopping was successfully implemented. Further application of value-based adaptive approaches could be useful to assess the efficiency of alternative study designs.

Strategic use of resources to enhance colorectal cancer screening for patients with diabetes (SURE: CRC4D) in federally qualified health centers: a protocol for hybrid type ii effectiveness-implementation trial

BACKGROUND

Persons with diabetes have 27% elevated risk of developing colorectal cancer (CRC) and are disproportionately from priority health disparities populations. Federally qualified health centers (FQHCs) struggle to implement CRC screening programs for average risk patients. Strategies to effectively prioritize and optimize CRC screening for patients with diabetes in the primary care safety-net are needed.

METHODS

Guided by the Exploration, Preparation, Implementation and Sustainment Framework, we conducted a stakeholder-engaged process to identify multi-level change objectives for implementing optimized CRC screening for patients with diabetes in FQHCs. To identify change objectives, an implementation planning group of stakeholders from FQHCs, safety-net screening programs, and policy implementers were assembled and met over a 7-month period. Depth interviews (n = 18-20) with key implementation actors were conducted to identify and refine the materials, methods and strategies needed to support an implementation plan across different FQHC contexts. The planning group endorsed the following multi-component implementation strategies: identifying clinic champions, development/distribution of patient educational materials, developing and implementing quality monitoring systems, and convening clinical meetings. To support clinic champions during the initial implementation phase, two learning collaboratives and bi-weekly virtual facilitation will be provided. In single group, hybrid type 2 effectiveness-implementation trial, we will implement and evaluate these strategies in a in six safety net clinics (n = 30 patients with diabetes per site). The primary clinical outcomes are: (1) clinic-level colonoscopy uptake and (2) overall CRC screening rates for patients with diabetes assessed at baseline and 12-months post-implementation. Implementation outcomes include provider and staff fidelity to the implementation plan, patient acceptability, and feasibility will be assessed at baseline and 12-months post-implementation.

DISCUSSION

Study findings are poised to inform development of evidence-based implementation strategies to be tested for scalability and sustainability in a future hybrid 2 effectiveness-implementation clinical trial. The research protocol can be adapted as a model to investigate the development of targeted cancer prevention strategies in additional chronically ill priority populations.

TRIAL REGISTRATION

This study was registered in ClinicalTrials.gov (NCT05785780) on March 27, 2023 (last updated October 21, 2023).

Making Drug Approval Decisions in the Face of Uncertainty: Cumulative Evidence versus Value of Information

BACKGROUND

The COVID-19 pandemic underscored the criticality and complexity of decision making for novel treatment approval and further research. Our study aims to assess potential decision-making methodologies, an evaluation vital for refining future public health crisis responses.

METHODS

We compared 4 decision-making approaches to drug approval and research: the Food and Drug Administration's policy decisions, cumulative meta-analysis, a prospective value-of-information (VOI) approach (using information available at the time of decision), and a reference standard (retrospective VOI analysis using information available in hindsight). Possible decisions were to reject, accept, provide emergency use authorization, or allow access to new therapies only in research settings. We used monoclonal antibodies provided to hospitalized COVID-19 patients as a case study, examining the evidence from September 2020 to December 2021 and focusing on each method's capacity to optimize health outcomes and resource allocation.

RESULTS

Our findings indicate a notable discrepancy between policy decisions and the reference standard retrospective VOI approach with expected losses up to $269 billion USD, suggesting suboptimal resource use during the wait for emergency use authorization. Relying solely on cumulative meta-analysis for decision making results in the largest expected loss, while the policy approach showed a loss up to $16 billion and the prospective VOI approach presented the least loss (up to $2 billion).

CONCLUSION

Our research suggests that incorporating VOI analysis may be particularly useful for research prioritization and treatment implementation decisions during pandemics. While the prospective VOI approach was favored in this case study, further studies should validate the ideal decision-making method across various contexts. This study's findings not only enhance our understanding of decision-making strategies during a health crisis but also provide a potential framework for future pandemic responses.

HIGHLIGHTS

This study reviews discrepancies between a reference standard (retrospective VOI, using hindsight information) and 3 conceivable real-time approaches to research-treatment decisions during a pandemic, suggesting suboptimal use of resources.Of all prospective decision-making approaches considered, VOI closely mirrored the reference standard, yielding the least expected value loss across our study timeline.This study illustrates the possible benefit of VOI results and the need for evidence accumulation accompanied by modeling in health technology assessment for emerging therapies.

Adaptive design trials in eating disorder research: A scoping review

OBJECTIVE

This scoping review sought to map the breadth of literature on the use of adaptive design trials in eating disorder research.

METHOD

A systematic literature search was conducted in Medline, Scopus, PsycInfo, Emcare, Econlit, CINAHL and ProQuest Dissertations and Theses. Articles were included if they reported on an intervention targeting any type of eating disorder (including anorexia nervosa, bulimia nervosa, binge-eating disorder, and other specified feeding or eating disorders), and employed the use of an adaptive design trial to evaluate the intervention. Two independent reviewers screened citations for inclusion, and data abstraction was performed by one reviewer and verified by a second.

RESULTS

We identified five adaptive design trials targeting anorexia nervosa, bulimia nervosa and binge-eating disorder conducted in the USA and Australia. All employed adaptive treatment arm switching based on early response to treatment and identified a priori stopping rules. None of the studies included value of information analysis to guide adaptive design decisions and none included lived experience perspectives.

DISCUSSION

The limited use of adaptive designs in eating disorder trials represents a missed opportunity to improve enrolment targets, attrition rates, treatment outcomes and trial efficiency. We outline the range of adaptive methodologies, how they could be applied to eating disorder research, and the specific operational and statistical considerations relevant to adaptive design trials.

PUBLIC SIGNIFICANCE

Adaptive design trials are increasingly employed as flexible, efficient alternatives to fixed trial designs, but they are not often used in eating disorder research. This first scoping review identified five adaptive design trials targeting anorexia nervosa, bulimia nervosa and binge-eating disorder that employed treatment arm switching adaptive methodology. We make recommendations on the use of adaptive design trials for future eating disorder trials.

Value of Information for Clinical Trial Design: The Importance of Considering All Relevant Comparators

Value of Information (VOI) analyses calculate the economic value that could be generated by obtaining further information to reduce uncertainty in a health economic decision model. VOI has been suggested as a tool for research prioritisation and trial design as it can highlight economically valuable avenues for future research. Recent methodological advances have made it increasingly feasible to use VOI in practice for research; however, there are critical differences between the VOI approach and the standard methods used to design research studies such as clinical trials. We aimed to highlight key differences between the research design approach based on VOI and standard clinical trial design methods, in particular the importance of considering the full decision context. We present two hypothetical examples to demonstrate that VOI methods are only accurate when (1) all feasible comparators are included in the decision model when designing research, and (2) all comparators are retained in the decision model once the data have been collected and a final treatment recommendation is made. Omitting comparators from either the design or analysis phase of research when using VOI methods can lead to incorrect trial designs and/or treatment recommendations. Overall, we conclude that incorrectly specifying the health economic model by ignoring potential comparators can lead to misleading VOI results and potentially waste scarce research resources.

A Guide to an Iterative Approach to Model-Based Decision Making in Health and Medicine: An Iterative Decision-Making Framework

Decision makers frequently face decisions about optimal resource allocation. A model-based economic evaluation can be used to guide decision makers in their choices by systematically evaluating the magnitude of expected health effects and costs of decision options and by making trade-offs explicit. We provide a guide to an iterative approach to the medical decision-making process by following a coherent framework, and outline the overarching iterative steps of model-based decision making. We systematized the framework by performing three steps. First, we compiled the existing guidelines provided by the ISPOR-SMDM Modeling Good Research Practices Task Force, and the ISPOR Value of Information Task Force. Second, we identified other previous work related to frameworks and guidelines for model-based decision analyses through a literature search in PubMed. Third, we assessed the role of the evidence and iterative process in decision making and formalized key steps in a model-based decision-making framework. We provide guidance on an iterative approach to medical decision making by applying the compiled iterative model-based decision-making framework. The framework formally combines the decision problem conceptualization (Part I), the model conceptualization and development (Part II), and the process of model-based decision analysis (Part III). Following the overarching steps of the framework ensures compliance to the principles of evidence-based medicine and regular updates of the evidence, given that value of information analysis represents an essential component of model-based decision analysis in the framework. Following the provided guide and the steps outlined in the framework can help inform various health care decisions, and therefore it has the potential to improve decision making.

Identifying the Optimum Strategy for Identifying Adults and Children With Celiac Disease: A Cost-Effectiveness and Value of Information Analysis

OBJECTIVES

Celiac disease (CD) is thought to affect around 1% of people in the United Kingdom, but only approximately 30% are diagnosed. The aim of this work was to assess the cost-effectiveness of strategies for identifying adults and children with CD in terms of who to test and which tests to use.

METHODS

A decision tree and Markov model were used to describe testing strategies and model long-term consequences of CD. The analysis compared a selection of pre-test probabilities of CD above which patients should be screened, as well as the use of different serological tests, with or without genetic testing. Value of information analysis was used to prioritize parameters for future research.

RESULTS

Using serological testing alone in adults, immunoglobulin A (IgA) tissue transglutaminase (tTG) at a 1% pre-test probability (equivalent to population screening) was most cost-effective. If combining serological testing with genetic testing, human leukocyte antigen combined with IgA tTG at a 5% pre-test probability was most cost-effective. In children, the most cost-effective strategy was a 10% pre-test probability with human leukocyte antigen plus IgA tTG. Value of information analysis highlighted the probability of late diagnosis of CD and the accuracy of serological tests as important parameters. The analysis also suggested prioritizing research in adult women over adult men or children.

CONCLUSIONS

For adults, these cost-effectiveness results suggest UK National Screening Committee Criteria for population-based screening for CD should be explored. Substantial uncertainty in the results indicate a high value in conducting further research.

Value of Information Analysis of a Web-Based Self-Management Intervention for Chronic Obstructive Pulmonary Disease

Objective: Technology-based programs can be cost-effective in the management of chronic obstructive pulmonary disease (COPD). However, cost-effectiveness estimates always contain some uncertainty, and decisions based upon them carry some risk. We conducted a value of information (VOI) analysis to estimate the value of additional research of a web-based self-management intervention for COPD to reduce the costs associated with uncertainty. Methods: We used a 10,000-iteration cost-effectiveness model from the health care payer perspective to calculate the expected value of perfect information (EVPI) at the patient- and population-level. An opportunity loss was incurred when the web-based intervention did not produce a greater net monetary benefit than usual care in an iteration. We calculated the probability of opportunity loss and magnitude of opportunity costs as a function of baseline health utility. We aggregated opportunity costs over the projected incident population of inpatient COPD patients over 10 years and estimated it as a function of the willingness-to-pay (WTP) threshold. Costs are in 2022 U.S. Dollars. Results: Opportunity losses were found in 22.7% of the iterations. The EVPIpatient was $78 per patient (95% confidence interval: $75-$82). The probability that the intervention was the optimal strategy varied across baseline health utilities. The EVPIpopulation was $506,666,882 over 10 years for a WTP of $50,000. Conclusions: Research estimated to cost up to $500 million would be warranted to reduce uncertainty. Future research could focus on identifying the impact of baseline health utilities to maximize the cost savings of the intervention. Other considerations for future research priorities include implementation efforts for technology-based interventions.

Estimating the Potential Benefits of Confirmatory Trials for Drugs with Accelerated Approval: A Comprehensive Value of Information Framework

BACKGROUND

The US Food and Drug Administration's Accelerated Approval (AA) policy provides a pathway for patients to access potentially life-saving drugs rapidly. However, the use of surrogate endpoints, single-arm designs, and small sample sizes in preliminary trials that support AAs can lead to uncertainty regarding the clinical benefits of such drugs. This study aims to develop a comprehensive value of information (VOI) framework for assessing the potential benefits of future confirmatory trials, accounting for the various uncertainties inherent in preliminary trials.

METHODS

I formulated an expected value of information from confirmatory trial (EVICT) metric, which evaluates the potential benefits of a confirmatory trial that would reduce those uncertainties by using a clinically meaningful endpoint, a randomized control, and increased sample size. The EVICT metric can quantify the expected benefits of a well-designed confirmatory trial or an inadequately designed one that continues to use surrogate endpoints or single-arm design. The framework was illustrated using a hypothetical AA drug for metastatic breast cancer.

RESULTS

The case study demonstrates that a highly uncertain preliminary trial of an AA drug was associated with a substantial EVICT. A confirmatory trial with an increased sample size for this AA drug, utilizing a clinically meaningful endpoint and randomized control, yielded a population-level EVICT of $12.6 million. Persistently using a surrogate endpoint and single-arm trial design would reduce the EVICT by 60%.

CONCLUSIONS

This framework can provide accurate VOI estimates to guide coverage policies, value-based pricing, and the design of confirmatory trials for AA drugs.

Consolidated Health Economic Evaluation Reporting Standards - Value of Information (CHEERS-VOI): Explanation and Elaboration

OBJECTIVES

Although the ISPOR Value of Information (VOI) Task Force's reports outline VOI concepts and provide good-practice recommendations, there is no guidance for reporting VOI analyses. VOI analyses are usually performed alongside economic evaluations for which the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 Statement provides reporting guidelines. Thus, we developed the CHEERS-VOI checklist to provide reporting guidance and checklist to support the transparent, reproducible, and high-quality reporting of VOI analyses.

METHODS

A comprehensive literature review generated a list of 26 candidate reporting items. These candidate items underwent a Delphi procedure with Delphi participants through 3 survey rounds. Participants rated each item on a 9-point Likert scale to indicate its relevance when reporting the minimal, essential information about VOI methods and provided comments. The Delphi results were reviewed at 2-day consensus meetings and the checklist was finalized using anonymous voting.

RESULTS

We had 30, 25, and 24 Delphi respondents in rounds 1, 2, and 3, respectively. After incorporating revisions recommended by the Delphi participants, all 26 candidate items proceeded to the 2-day consensus meetings. The final CHEERS-VOI checklist includes all CHEERS items, but 7 items require elaboration when reporting VOI. Further, 6 new items were added to report information relevant only to VOI (eg, VOI methods applied).

CONCLUSIONS

The CHEERS-VOI checklist should be used when a VOI analysis is performed alongside economic evaluations. The CHEERS-VOI checklist will help decision makers, analysts and peer reviewers in the assessment and interpretation of VOI analyses and thereby increase transparency and rigor in decision making.

Alternative cascade-testing protocols for identifying and managing patients with familial hypercholesterolaemia: systematic reviews, qualitative study and cost-effectiveness analysis

Background

Cascade testing the relatives of people with familial hypercholesterolaemia is an efficient approach to identifying familial hypercholesterolaemia. The cascade-testing protocol starts with identifying an index patient with familial hypercholesterolaemia, followed by one of three approaches to contact other relatives: indirect approach, whereby index patients contact their relatives; direct approach, whereby the specialist contacts the relatives; or a combination of both direct and indirect approaches. However, it is unclear which protocol may be most effective.

Objectives

The objectives were to determine the yield of cases from different cascade-testing protocols, treatment patterns, and short- and long-term outcomes for people with familial hypercholesterolaemia; to evaluate the cost-effectiveness of alternative protocols for familial hypercholesterolaemia cascade testing; and to qualitatively assess the acceptability of different cascade-testing protocols to individuals and families with familial hypercholesterolaemia, and to health-care providers.

Design and methods

This study comprised systematic reviews and analysis of three data sets: PASS (PASS Software, Rijswijk, the Netherlands) hospital familial hypercholesterolaemia databases, the Clinical Practice Research Datalink (CPRD)-Hospital Episode Statistics (HES) linked primary-secondary care data set, and a specialist familial hypercholesterolaemia register. Cost-effectiveness modelling, incorporating preceding analyses, was undertaken. Acceptability was examined in interviews with patients, relatives and health-care professionals.

Result

Systematic review of protocols: based on data from 4 of the 24 studies, the combined approach led to a slightly higher yield of relatives tested [40%, 95% confidence interval (CI) 37% to 42%] than the direct (33%, 95% CI 28% to 39%) or indirect approaches alone (34%, 95% CI 30% to 37%). The PASS databases identified that those contacted directly were more likely to complete cascade testing (p < 0.01); the CPRD-HES data set indicated that 70% did not achieve target treatment levels, and demonstrated increased cardiovascular disease risk among these individuals, compared with controls (hazard ratio 9.14, 95% CI 8.55 to 9.76). The specialist familial hypercholesterolaemia register confirmed excessive cardiovascular morbidity (standardised morbidity ratio 7.17, 95% CI 6.79 to 7.56). Cost-effectiveness modelling found a net health gain from diagnosis of -0.27 to 2.51 quality-adjusted life-years at the willingness-to-pay threshold of £15,000 per quality-adjusted life-year gained. The cost-effective protocols cascaded from genetically confirmed index cases by contacting first- and second-degree relatives simultaneously and directly. Interviews found a service-led direct-contact approach was more reliable, but combining direct and indirect approaches, guided by index patients and family relationships, may be more acceptable.

Limitations

Systematic reviews were not used in the economic analysis, as relevant studies were lacking or of poor quality. As only a proportion of those with primary care-coded familial hypercholesterolaemia are likely to actually have familial hypercholesterolaemia, CPRD analyses are likely to underestimate the true effect. The cost-effectiveness analysis required assumptions related to the long-term cardiovascular disease risk, the effect of treatment on cholesterol and the generalisability of estimates from the data sets. Interview recruitment was limited to white English-speaking participants.

Conclusions

Based on limited evidence, most cost-effective cascade-testing protocols, diagnosing most relatives, select index cases by genetic testing, with services directly contacting relatives, and contacting second-degree relatives even if first-degree relatives have not been tested. Combined approaches to contact relatives may be more suitable for some families.

Future work

Establish a long-term familial hypercholesterolaemia cohort, measuring cholesterol levels, treatment and cardiovascular outcomes. Conduct a randomised study comparing different approaches to contact relatives.

Study registration

This study is registered as PROSPERO CRD42018117445 and CRD42019125775.

Funding

This project was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 27, No. 16. See the NIHR Journals Library website for further project information.

The Value-of-Information and Value-of-Implementation from Clinical Trials of Diagnostic Tests for HIV-Associated Tuberculosis: A Modeling Analysis

Objectives. Conventional value-of-information (VOI) analysis assumes complete uptake of an optimal decision. We employed an extended framework that includes value-of-implementation (VOM)-the benefit of encouraging adoption of an optimal strategy-and estimated how future trials of diagnostic tests for HIV-associated tuberculosis could improve public health decision making and clinical and economic outcomes. Methods. We evaluated the clinical outcomes and costs, given current information, of 3 tuberculosis screening strategies among hospitalized people with HIV in South Africa: sputum Xpert (Xpert), sputum Xpert plus urine AlereLAM (Xpert+AlereLAM), and sputum Xpert plus the newer, more sensitive, and costlier urine FujiLAM (Xpert+FujiLAM). We projected the incremental net monetary benefit (INMB) of decision making based on results of a trial comparing mortality with each strategy, rather than decision making based solely on current knowledge of FujiLAM's improved diagnostic performance. We used a validated microsimulation to estimate VOI (the INMB of reducing parameter uncertainty before decision making) and VOM (the INMB of encouraging adoption of an optimal strategy). Results. With current information, adopting Xpert+FujiLAM yields 0.4 additional life-years/person compared with current practices (assumed 50% Xpert and 50% Xpert+AlereLAM). While the decision to adopt this optimal strategy is unaffected by information from the clinical trial (VOI = $ 0 at $3,000/year-of-life saved willingness-to-pay threshold), there is value in scaling up implementation of Xpert+FujiLAM, which results in an INMB (representing VOM) of $650 million over 5 y. Conclusions. Conventional VOI methods account for the value of switching to a new optimal strategy based on trial data but fail to account for the persuasive value of trials in increasing uptake of the optimal strategy. Evaluation of trials should include a focus on their value in reducing barriers to implementation.

Highlights

In conventional VOI analysis, it is assumed that the optimal decision will always be adopted even without a trial. This can potentially lead to an underestimation of the value of trials when adoption requires new clinical trial evidence. To capture the influence that a trial may have on decision makers' willingness to adopt the optimal decision, we also consider value-of-implementation (VOM), a metric quantifying the benefit of new study information in promoting wider adoption of the optimal strategy. The overall value-of-a-trial (VOT) includes both VOI and VOM.Our model-based analysis suggests that the information obtained from a trial of screening strategies for HIV-associated tuberculosis in South Africa would have no value, when measured using traditional methods of VOI assessment. A novel strategy, which includes the urine FujiLAM test, is optimal from a health economic standpoint but is underutilized. A trial would reduce uncertainties around downstream health outcomes but likely would not change the optimal decision. The high VOT (nearly $700 million over 5 y) lies solely in promoting uptake of FujiLAM, represented as VOM.Our results highlight the importance of employing a more comprehensive approach for evaluating prospective trials, as conventional VOI methods can vastly underestimate their value. Trialists and funders can and should assess the VOT metric instead when considering trial designs and costs. If VOI is low, the VOM and cost of a trial can be compared with the benefits and costs of other outreach programs to determine the most cost-effective way to improve uptake.

Evaluating high-cost technologies - no need to throw the baby out with the bathwater

INTRODUCTION

Evidence generation for the health technology assessment (HTA) of a new technology is a long and expensive process with no guarantees that the health technology will be adopted and implemented into a health-care system. This would suggest that there is a greater risk of failure for a company developing a high-cost technology and therefore incentives (such as increasing the funding available for research or additional market exclusivity) may be needed to encourage development of such technologies as has been seen with many high-cost orphan drugs.

AREAS COVERED

This paper discusses some of the key issues relating to the evaluation of high-cost technologies through the use of existing HTA processes and what the challenges will be going forward.

EXPERT OPINION

We propose that while the current HTA process is robust, its evolution into accommodating the incorporation of real-world data and evidence alongside a life-cycle HTA approach should better enable developers to produce the evidence required on effectiveness and cost-effectiveness. This should lead to reduced decision uncertainty for HTA agencies to make adoption decisions in a more timely and efficient manner. Furthermore, budget impact analysis remains important in understanding the actual financial impact on health-care systems and budgets outside of the cost-effectiveness framework used to aid decision-making.

Guidance on the use of complex systems models for economic evaluations of public health interventions

To help health economic modelers respond to demands for greater use of complex systems models in public health. To propose identifiable features of such models and support researchers to plan public health modeling projects using these models. A working group of experts in complex systems modeling and economic evaluation was brought together to develop and jointly write guidance for the use of complex systems models for health economic analysis. The content of workshops was informed by a scoping review. A public health complex systems model for economic evaluation is defined as a quantitative, dynamic, non-linear model that incorporates feedback and interactions among model elements, in order to capture emergent outcomes and estimate health, economic and potentially other consequences to inform public policies. The guidance covers: when complex systems modeling is needed; principles for designing a complex systems model; and how to choose an appropriate modeling technique. This paper provides a definition to identify and characterize complex systems models for economic evaluations and proposes guidance on key aspects of the process for health economics analysis. This document will support the development of complex systems models, with impact on public health systems policy and decision making.

Comprehensive Review of Methods to Assess Uncertainty in Health Economic Evaluations

Uncertainty assessment is a cornerstone in model-based health economic evaluations (HEEs) that inform reimbursement decisions. No comprehensive overview of available uncertainty assessment methods currently exists. We aimed to review methods for uncertainty assessment for use in model-based HEEs, by conducting a snowballing review. We categorised all methods according to their stage of use relating to uncertainty assessment (identification, analysis, communication). Additionally, we classified identification methods according to sources of uncertainty, and subdivided analysis and communication methods according to their purpose. The review identified a total of 80 uncertainty methods: 30 identification, 28 analysis, and 22 communication methods. Uncertainty identification methods exist to address uncertainty from different sources. Most identification methods were developed with the objective to assess related concepts such as validity, model quality, and relevance. Almost all uncertainty analysis and communication methods required uncertainty to be quantified and inclusion of uncertainties in probabilistic analysis. Our review can help analysts and decision makers in selecting uncertainty assessment methods according to their aim and purpose of the assessment. We noted a need for further clarification of terminology and guidance on the use of (combinations of) methods to identify uncertainty and related concepts such as validity and quality. A key finding is that uncertainty assessment relies heavily on quantification, which may necessitate increased use of expert elicitation and/or the development of methods to assess unquantified uncertainty.

Protocol for a bandit-based response adaptive trial to evaluate the effectiveness of brief self-guided digital interventions for reducing psychological distress in university students: the Vibe Up study

INTRODUCTION

Meta-analytical evidence confirms a range of interventions, including mindfulness, physical activity and sleep hygiene, can reduce psychological distress in university students. However, it is unclear which intervention is most effective. Artificial intelligence (AI)-driven adaptive trials may be an efficient method to determine what works best and for whom. The primary purpose of the study is to rank the effectiveness of mindfulness, physical activity, sleep hygiene and an active control on reducing distress, using a multiarm contextual bandit-based AI-adaptive trial method. Furthermore, the study will explore which interventions have the largest effect for students with different levels of baseline distress severity.

METHODS AND ANALYSIS

The Vibe Up study is a pragmatically oriented, decentralised AI-adaptive group sequential randomised controlled trial comparing the effectiveness of one of three brief, 2-week digital self-guided interventions (mindfulness, physical activity or sleep hygiene) or active control (ecological momentary assessment) in reducing self-reported psychological distress in Australian university students. The adaptive trial methodology involves up to 12 sequential mini-trials that allow for the optimisation of allocation ratios. The primary outcome is change in psychological distress (Depression, Anxiety and Stress Scale, 21-item version, DASS-21 total score) from preintervention to postintervention. Secondary outcomes include change in physical activity, sleep quality and mindfulness from preintervention to postintervention. Planned contrasts will compare the four groups (ie, the three intervention and control) using self-reported psychological distress at prespecified time points for interim analyses. The study aims to determine the best performing intervention, as well as ranking of other interventions.

ETHICS AND DISSEMINATION

Ethical approval was sought and obtained from the UNSW Sydney Human Research Ethics Committee (HREC A, HC200466). A trial protocol adhering to the requirements of the Guideline for Good Clinical Practice was prepared for and approved by the Sponsor, UNSW Sydney (Protocol number: HC200466_CTP).

TRIAL REGISTRATION NUMBER

ACTRN12621001223820.

Information Density in Decision Analysis

Information value has been proposed and used as a probabilistic sensitivity measure, the idea being that uncertain parameters having higher information value are precisely those to which an optimal decision is more sensitive. In this paper, we study the notion of information density as a graphical complement to information value analysis, one that augments an information value calculation with associated directions of information gain. We formally examine mathematical details absent from its earlier presentation that guarantee information density exists and is well posed and describe its relationship to alternate measures of information value. We present its application in the context of a realistic case study and discuss the associated insights.

The impact of statistical properties of incremental monetary net benefit and incremental cost-effectiveness ratio on health economic modeling choices

INTRODUCTION

There is controversy on whether to use incremental monetary net benefit (INMB) or incremental cost-effectiveness ratio (ICER) in health economic evaluations alongside randomized controlled trials. We studied the impact of restricted mean survival time (RMST) on the long-term projection of INMB and ICER.

METHODS

We analyzed the unbiasedness and efficiency of ICER and INMB by (1) deriving the metrics' expected values and variances based on theoretical probability distributions, (2) simulating their 15-year post-trial projections based on between-arm-RMST-gained through a 2 × 4 × 2 factorial experiment of Markov 2-state microsimulations. Simulations and comparison were run on the data from the Cardiovascular Outcomes for People Using Anticoagulation Strategies Study (COMPASS).

RESULTS

Our simulation findings using RMST showed that ICER was more efficient than INMB, regardless of disease populations, time horizon, modeling choices, and underlying probability distributions of incremental mean cost and effect. ICER had a small variance and thus showed its robustness to the choices of models.

CONCLUSION

INMB's variance varies with a willingness-to-pay (WTP) threshold quadratically while ICER's variance with a WTP threshold value quadratically while ICER's variance with incremental-mean-cost quadratically. A simple and naïve model can sufficiently estimate ICER. Future metrics are expected to be health-economic-meaningful, unambiguous, unbiased, efficient, and statistical-inference-friendly.

Cost-Effectiveness and Budget Impact of Future Developments With Whole-Genome Sequencing for Patients With Lung Cancer

OBJECTIVES

This study aimed to investigate the cost-effectiveness, budget impact (BI), and impact of uncertainty of future developments concerning whole-genome sequencing (WGS) as a clinical diagnostic test compared with standard of care (SoC) in patients with locally advanced and metastatic non-small cell lung cancer.

METHODS

A total of 3 likely scenarios to take place within 5 years (according to experts) were simulated using a previously developed, peer reviewed, and published decision model. The scenarios concerned "WGS results used for treatment selection" (scenario 1), "WGS-based biomarker for immunotherapy" (scenario 2), and "off-label drug approval for WGS results" (scenario 3). Two diagnostic strategies of the original model, "SoC" and "WGS as a diagnostic test" (base model), were used to compare our scenarios with. Outcomes were reported for the base model, all scenarios separately, combined (combined unweighted), and weighted by likelihood (combined weighted). Cost-effectiveness, BI, and value of information analyses were performed for WGS compared with SoC.

RESULTS

Total costs and quality-adjusted life-years for SoC in metastatic non-small cell lung cancer were €149 698 and 1.235. Incremental outcomes of WGS were €1529/0.002(base model), -€222/0.020(scenario 1), -€2576/0.023(scenario 2), €388/0.024(scenario 3), -€5041/0.060(combined unweighted), and -€1715/0.029(combined weighted). The annual BI for adopting WGS for this population in The Netherlands ranged between €682 million (combined unweighted) and €714 million (base model). The consequences of uncertainty amounted to €3.4 million for all scenarios (combined weighted) and to €699 000 for the diagnostic yield of WGS alone (combined weighted).

CONCLUSIONS

Our findings suggest that it is likely for WGS to become cost-effective within the near future if it identifies more patients with actionable targets and show the impact of uncertainty regarding its diagnostic yield. Modeling future scenarios can be useful to consider early adoption of WGS while timely anticipating on unforeseen developments before final conclusions are reached.

Extensions of Health Economic Evaluations in R for Microsoft Excel Users: A Tutorial for Incorporating Heterogeneity and Conducting Value of Information Analyses

Advanced health economic analysis techniques currently performed in Microsoft Excel, such as incorporating heterogeneity, time-dependent transitions and a value of information analysis, can be easily transferred to R. Often the outputs of survival analyses (such as Weibull regression models) will estimate the impacts of correlated patient characteristics on patient outcomes, and are utilised directly as inputs for health economic decision models. This tutorial provides a step-by-step guide of how to conduct such analyses with a Markov model developed in R, and offers a comparison with established analyses performed in Microsoft Excel. This is done through the conversion of a previously published Microsoft Excel case study of a hip replacement surgery cost-effectiveness model. We hope that this paper can act as a facilitator in switching decision models from Microsoft Excel to R for complex health economic analyses, providing open-access code and data, suitable for future adaptation.

Approaches to prioritising research for clinical trial networks: a scoping review

BACKGROUND

Prioritisation of clinical trials ensures that the research conducted meets the needs of stakeholders, makes the best use of resources and avoids duplication. The aim of this review was to identify and critically appraise approaches to research prioritisation applicable to clinical trials, to inform best practice guidelines for clinical trial networks and funders.

METHODS

A scoping review of English-language published literature and research organisation websites (January 2000 to January 2020) was undertaken to identify primary studies, approaches and criteria for research prioritisation. Data were extracted and tabulated, and a narrative synthesis was employed.

RESULTS

Seventy-eight primary studies and 18 websites were included. The majority of research prioritisation occurred in oncology and neurology disciplines. The main reasons for prioritisation were to address a knowledge gap (51 of 78 studies [65%]) and to define patient-important topics (28 studies, [35%]). In addition, research organisations prioritised in order to support their institution's mission, invest strategically, and identify best return on investment. Fifty-seven of 78 (73%) studies used interpretative prioritisation approaches (including Delphi surveys, James Lind Alliance and consensus workshops); six studies used quantitative approaches (8%) such as prospective payback or value of information (VOI) analyses; and 14 studies used blended approaches (18%) such as nominal group technique and Child Health Nutritional Research Initiative. Main criteria for prioritisation included relevance, appropriateness, significance, feasibility and cost-effectiveness.

CONCLUSION

Current research prioritisation approaches for groups conducting and funding clinical trials are largely interpretative. There is an opportunity to improve the transparency of prioritisation through the inclusion of quantitative approaches.

Estimated Monetary Value of Future Research Clarifying Uncertainties Around the Optimal Adult Hearing Screening Schedule

Importance

Adult hearing screening is not routinely performed, and most individuals with hearing loss (HL) have never had their hearing tested as adults.

Objective

To project the monetary value of future research clarifying uncertainties around the optimal adult hearing screening schedule.

Design, Setting, and Participants

In this economic evaluation, a validated decision model of HL (DeciBHAL-US: Decision model of the Burden of Hearing loss Across the Lifespan) was used to simulate current detection and treatment of HL vs hearing screening schedules. Key model inputs included HL incidence (0.06%-10.42%/y), hearing aid uptake (0.54%-8.14%/y), screening effectiveness (1.62 × hearing aid uptake), utility benefits of hearing aids (+0.11), and hearing aid device costs ($3690). Distributions to model parameters for probabilistic uncertainty analysis were assigned. The expected value of perfect information (EVPI) and expected value of partial perfect information (EVPPI) using a willingness to pay of $100 000 per quality-adjusted life-year (QALY) was estimated. The EVPI and EVPPI estimate the upper bound of the dollar value of future research. This study was based on 40-year-old persons over their remaining lifetimes in a US primary care setting.

Exposures

Screening schedules beginning at ages 45, 55, 65, and 75 years, and frequencies of every 1 or 5 years.

Main Outcomes and Measures

The main outcomes were QALYs and costs (2020 US dollars) from a health system perspective.

Results

The average incremental cost-effectiveness ratio for yearly screening beginning at ages 55 to 75 years ranged from $39 200 to $80 200/QALY. Yearly screening beginning at age 55 years was the optimal screening schedule in 38% of probabilistic uncertainty analysis simulations. The population EVPI, or value of reducing all uncertainty, was $8.2 to $12.6 billion varying with willingness to pay and the EVPPI, or value of reducing all screening effectiveness uncertainty, was $2.4 billion.

Conclusions and Relevance

In this economic evaluation of US adult hearing screening, large uncertainty around the optimal adult hearing screening schedule was identified. Future research on hearing screening has a high potential value so is likely justified.

Defining the optimum strategy for identifying adults and children with coeliac disease: systematic review and economic modelling

BACKGROUND

Coeliac disease is an autoimmune disorder triggered by ingesting gluten. It affects approximately 1% of the UK population, but only one in three people is thought to have a diagnosis. Untreated coeliac disease may lead to malnutrition, anaemia, osteoporosis and lymphoma.

OBJECTIVES

The objectives were to define at-risk groups and determine the cost-effectiveness of active case-finding strategies in primary care.

DESIGN

(1) Systematic review of the accuracy of potential diagnostic indicators for coeliac disease. (2) Routine data analysis to develop prediction models for identification of people who may benefit from testing for coeliac disease. (3) Systematic review of the accuracy of diagnostic tests for coeliac disease. (4) Systematic review of the accuracy of genetic tests for coeliac disease (literature search conducted in April 2021). (5) Online survey to identify diagnostic thresholds for testing, starting treatment and referral for biopsy. (6) Economic modelling to identify the cost-effectiveness of different active case-finding strategies, informed by the findings from previous objectives.

DATA SOURCES

For the first systematic review, the following databases were searched from 1997 to April 2021: MEDLINE® (National Library of Medicine, Bethesda, MD, USA), Embase® (Elsevier, Amsterdam, the Netherlands), Cochrane Library, Web of Science™ (Clarivate™, Philadelphia, PA, USA), the World Health Organization International Clinical Trials Registry Platform ( WHO ICTRP ) and the National Institutes of Health Clinical Trials database. For the second systematic review, the following databases were searched from January 1990 to August 2020: MEDLINE, Embase, Cochrane Library, Web of Science, Kleijnen Systematic Reviews ( KSR ) Evidence, WHO ICTRP and the National Institutes of Health Clinical Trials database. For prediction model development, Clinical Practice Research Datalink GOLD, Clinical Practice Research Datalink Aurum and a subcohort of the Avon Longitudinal Study of Parents and Children were used; for estimates for the economic models, Clinical Practice Research Datalink Aurum was used.

REVIEW METHODS

For review 1, cohort and case-control studies reporting on a diagnostic indicator in a population with and a population without coeliac disease were eligible. For review 2, diagnostic cohort studies including patients presenting with coeliac disease symptoms who were tested with serological tests for coeliac disease and underwent a duodenal biopsy as reference standard were eligible. In both reviews, risk of bias was assessed using the quality assessment of diagnostic accuracy studies 2 tool. Bivariate random-effects meta-analyses were fitted, in which binomial likelihoods for the numbers of true positives and true negatives were assumed.

RESULTS

People with dermatitis herpetiformis, a family history of coeliac disease, migraine, anaemia, type 1 diabetes, osteoporosis or chronic liver disease are 1.5-2 times more likely than the general population to have coeliac disease; individual gastrointestinal symptoms were not useful for identifying coeliac disease. For children, women and men, prediction models included 24, 24 and 21 indicators of coeliac disease, respectively. The models showed good discrimination between patients with and patients without coeliac disease, but performed less well when externally validated. Serological tests were found to have good diagnostic accuracy for coeliac disease. Immunoglobulin A tissue transglutaminase had the highest sensitivity and endomysial antibody the highest specificity. There was little improvement when tests were used in combination. Survey respondents (n = 472) wanted to be 66% certain of the diagnosis from a blood test before starting a gluten-free diet if symptomatic, and 90% certain if asymptomatic. Cost-effectiveness analyses found that, among adults, and using serological testing alone, immunoglobulin A tissue transglutaminase was most cost-effective at a 1% pre-test probability (equivalent to population screening). Strategies using immunoglobulin A endomysial antibody plus human leucocyte antigen or human leucocyte antigen plus immunoglobulin A tissue transglutaminase with any pre-test probability had similar cost-effectiveness results, which were also similar to the cost-effectiveness results of immunoglobulin A tissue transglutaminase at a 1% pre-test probability. The most practical alternative for implementation within the NHS is likely to be a combination of human leucocyte antigen and immunoglobulin A tissue transglutaminase testing among those with a pre-test probability above 1.5%. Among children, the most cost-effective strategy was a 10% pre-test probability with human leucocyte antigen plus immunoglobulin A tissue transglutaminase, but there was uncertainty around the most cost-effective pre-test probability. There was substantial uncertainty in economic model results, which means that there would be great value in conducting further research.

LIMITATIONS

The interpretation of meta-analyses was limited by the substantial heterogeneity between the included studies, and most included studies were judged to be at high risk of bias. The main limitations of the prediction models were that we were restricted to diagnostic indicators that were recorded by general practitioners and that, because coeliac disease is underdiagnosed, it is also under-reported in health-care data. The cost-effectiveness model is a simplification of coeliac disease and modelled an average cohort rather than individuals. Evidence was weak on the probability of routine coeliac disease diagnosis, the accuracy of serological and genetic tests and the utility of a gluten-free diet.

CONCLUSIONS

Population screening with immunoglobulin A tissue transglutaminase (1% pre-test probability) and of immunoglobulin A endomysial antibody followed by human leucocyte antigen testing or human leucocyte antigen testing followed by immunoglobulin A tissue transglutaminase with any pre-test probability appear to have similar cost-effectiveness results. As decisions to implement population screening cannot be made based on our economic analysis alone, and given the practical challenges of identifying patients with higher pre-test probabilities, we recommend that human leucocyte antigen combined with immunoglobulin A tissue transglutaminase testing should be considered for adults with at least a 1.5% pre-test probability of coeliac disease, equivalent to having at least one predictor. A more targeted strategy of 10% pre-test probability is recommended for children (e.g. children with anaemia).

FUTURE WORK

Future work should consider whether or not population-based screening for coeliac disease could meet the UK National Screening Committee criteria and whether or not it necessitates a long-term randomised controlled trial of screening strategies. Large prospective cohort studies in which all participants receive accurate tests for coeliac disease are needed.

STUDY REGISTRATION

This study is registered as PROSPERO CRD42019115506 and CRD42020170766.

FUNDING

This project was funded by the National Institute for Health and Care Research ( NIHR ) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 26, No. 44. See the NIHR Journals Library website for further project information.

Challenges around quantifying uncertainty in a holistic approach to hard-to-heal wound management: Health economic perspective

Treatment of hard-to-heal wounds involves a holistic approach for choosing between available treatment options. However, evidence for informing these choices is sparse, introducing uncertainty into decisions about the optimum treatment pathways that reflect the vast heterogeneity in this patient population. This paper discusses the existing clinical and health economic literature in order to provide insight into sources of uncertainty in the evaluation of the holistic approach to management of the hard-to-heal wounds, and how this uncertainty can be appropriately reflected in research. We identified three key sources of uncertainty in the evaluation of chronic wound treatments, namely heterogeneity in aetiology and patient populations, heterogeneity in treatment pathways, and challenges around capturing all relevant outcomes. Reflecting these complexities requires sophisticated modelling of treatment sequencing and long-term outcomes. The paper discusses how the scope specification, scenario analyses, and sensitivity analyses can be used to fully characterise analytical uncertainty.

Coverage with evidence development for medical devices in Europe: Can practice meet theory?

Health economists have written extensively on the design and implementation of coverage with evidence development (CED) schemes and have proposed theoretical frameworks based on cost-effectiveness modeling and value of information analysis. CED may aid decision-makers when there is uncertainty about the (cost-)effectiveness of a new health technology at the time of reimbursement. Medical devices are potential candidates for CED schemes, as regulatory regimes do not usually require the same level of efficacy and safety data normally needed for pharmaceuticals. The purpose of this research is to assess whether the actual practice of CED for medical devices in Europe meets the theoretical principles proposed by health economists and whether theory and practice can be more closely aligned. Based on decision-makers' perceptions of the challenges associated with CED schemes, plus examples from the schemes themselves, we discuss a series of proposals for assessing the desirability of schemes, their design, implementation, and evaluation. These proposals, while reflecting the practical challenges with developing CED programs, embody many of the principles suggested by economists and should support decision-makers in dealing with uncertainty about the real-world performance of devices.

Exploring the misalignment on the value of further research between payers and manufacturers. A case study on a novel total artificial heart

Payers and manufacturers can disagree on the appropriate level of evidence that is required for new medical devices, resulting in high societal costs due to decisions taken with sub-optimal information. A cost-effectiveness model of a hypothetical total artificial heart was built using data from the literature and the (simulated) results of a pivotal study. The expected value of perfect information (EVPI) was calculated from both the payer and manufacturer perspectives, using net monetary benefit and the company's return on investment respectively. A function was also defined, linking effectiveness to market shares. Additional constraints such as a minimum clinical difference or maximum budget impact were introduced into the company's decisions to simulate additional barriers to adoption. The difference in the EVPI between manufacturers and payers varied greatly depending on the underlying decision rules and constraints. The manufacturer's EVPI depends on the probability of being reimbursed, the uncertainty on the (cost-)effectiveness of the technology, as well as other parameters relating to initial investments, operating costs and market dynamics. The use of Value of information for both perspectives can outline potential misalignments and can be particularly useful to inform early dialogs between manufacturers and payers, or negotiations on conditional reimbursement schemes.

Practical guidance for planning resources required to support publicly-funded adaptive clinical trials

Adaptive designs are a class of methods for improving efficiency and patient benefit of clinical trials. Although their use has increased in recent years, research suggests they are not used in many situations where they have potential to bring benefit. One barrier to their more widespread use is a lack of understanding about how the choice to use an adaptive design, rather than a traditional design, affects resources (staff and non-staff) required to set-up, conduct and report a trial. The Costing Adaptive Trials project investigated this issue using quantitative and qualitative research amongst UK Clinical Trials Units. Here, we present guidance that is informed by our research, on considering the appropriate resourcing of adaptive trials. We outline a five-step process to estimate the resources required and provide an accompanying costing tool. The process involves understanding the tasks required to undertake a trial, and how the adaptive design affects them. We identify barriers in the publicly funded landscape and provide recommendations to trial funders that would address them. Although our guidance and recommendations are most relevant to UK non-commercial trials, many aspects are relevant more widely.

A Conceptual Framework for Life-Cycle Health Technology Assessment

OBJECTIVES

Health technology assessment (HTA) uses evidence appraisal and synthesis with economic evaluation to inform adoption decisions. Standard HTA processes sometimes struggle to (1) support decisions that involve significant uncertainty and (2) encourage continued generation of and adaptation to new evidence. We propose the life-cycle (LC)-HTA framework, addressing these challenges by providing additional tools to decision makers and improving outcomes for all stakeholders.

METHODS

Under the LC-HTA framework, HTA processes align to LC management. LC-HTA introduces changes in HTA methods to minimize analytic time while optimizing decision certainty. Where decision uncertainty exists, we recommend risk-based pricing and research-oriented managed access (ROMA). Contractual procurement agreements define the terms of reassessment and provide additional decision options to HTA agencies. LC-HTA extends value-of-information methods to inform ROMA agreements, leveraging routine, administrative data, and registries to reduce uncertainty.

RESULTS

LC-HTA enables the adoption of high-value high-risk innovations while improving health system sustainability through risk-sharing and reducing uncertainty. Responsiveness to evolving evidence is improved through contractually embedded decision rules to simplify reassessment. ROMA allows conditional adoption to obtain additional information, with confidence that the net value of that adoption decision is positive.

CONCLUSIONS

The LC-HTA framework improves outcomes for patients, sponsors, and payers. Patients benefit through earlier access to new technologies. Payers increase the value of the technologies they invest in and gain mechanisms to review investments. Sponsors benefit through greater certainty in outcomes related to their investment, swifter access to markets, and greater opportunities to demonstrate value.

An Efficient Method for Computing Expected Value of Sample Information for Survival Data from an Ongoing Trial

BACKGROUND

Decisions about new health technologies are increasingly being made while trials are still in an early stage, which may result in substantial uncertainty around key decision drivers such as estimates of life expectancy and time to disease progression. Additional data collection can reduce uncertainty, and its value can be quantified by computing the expected value of sample information (EVSI), which has typically been described in the context of designing a future trial. In this article, we develop new methods for computing the EVSI of extending an existing trial's follow-up, first for an assumed survival model and then extending to capture uncertainty about the true survival model.

METHODS

We developed a nested Markov Chain Monte Carlo procedure and a nonparametric regression-based method. We compared the methods by computing single-model and model-averaged EVSI for collecting additional follow-up data in 2 synthetic case studies.

RESULTS

There was good agreement between the 2 methods. The regression-based method was fast and straightforward to implement, and scales easily to include any number of candidate survival models in the model uncertainty case. The nested Monte Carlo procedure, on the other hand, was extremely computationally demanding when we included model uncertainty.

CONCLUSIONS

We present a straightforward regression-based method for computing the EVSI of extending an existing trial's follow-up, both where a single known survival model is assumed and where we are uncertain about the true survival model. EVSI for ongoing trials can help decision makers determine whether early patient access to a new technology can be justified on the basis of the current evidence or whether more mature evidence is needed.

HIGHLIGHTS

Decisions about new health technologies are increasingly being made while trials are still in an early stage, which may result in substantial uncertainty around key decision drivers such as estimates of life-expectancy and time to disease progression. Additional data collection can reduce uncertainty, and its value can be quantified by computing the expected value of sample information (EVSI), which has typically been described in the context of designing a future trial.In this article, we have developed new methods for computing the EVSI of extending a trial's follow-up, both where a single known survival model is assumed and where we are uncertain about the true survival model. We extend a previously described nonparametric regression-based method for computing EVSI, which we demonstrate in synthetic case studies is fast, straightforward to implement, and scales easily to include any number of candidate survival models in the EVSI calculations.The EVSI methods that we present in this article can quantify the need for collecting additional follow-up data before making an adoption decision given any decision-making context.

Methods for Communicating the Impact of Parameter Uncertainty in a Multiple-Strategies Cost-Effectiveness Comparison

Purpose

Analyzing and communicating uncertainty is essential in medical decision making. To judge whether risks are acceptable, policy makers require information on the expected outcomes but also on the uncertainty and potential losses related to the chosen strategy. We aimed to compare methods used to represent the impact of uncertainty in decision problems involving many strategies, enhance existing methods, and provide an open-source and easy-to-use tool.

Methods

We conducted a systematic literature search to identify methods used to represent the impact of uncertainty in cost-effectiveness analyses comparing multiple strategies. We applied the identified methods to probabilistic sensitivity analysis outputs of 3 published decision-analytic models comparing multiple strategies. Subsequently, we compared the following characteristics: type of information conveyed, use of a fixed or flexible willingness-to-pay threshold, output interpretability, and the graphical discriminatory ability. We further proposed adjustments and integration of methods to overcome identified limitations of existing methods.

Results

The literature search resulted in the selection of 9 methods. The 3 methods with the most favorable characteristics to compare many strategies were 1) the cost-effectiveness acceptability curve (CEAC) and cost-effectiveness acceptability frontier (CEAF), 2) the expected loss curve (ELC), and 3) the incremental benefit curve (IBC). The information required to assess confidence in a decision often includes the average loss and the probability of cost-effectiveness associated with each strategy. Therefore, we proposed the integration of information presented in an ELC and CEAC into a single heat map.

Conclusions

This article presents an overview of methods presenting uncertainty in multiple-strategy cost-effectiveness analyses, with their strengths and shortcomings. We proposed a heat map as an alternative method that integrates all relevant information required for health policy and medical decision making.

Highlights

A cost-effectiveness and value of information analysis to inform future research of tranexamic acid for older adults experiencing mild traumatic brain injury

BACKGROUND

Tranexamic acid reduces head injury deaths in patients with CT scan evidence of intracranial bleeding after mild traumatic brain injury (TBI). However, the cost-effectiveness of tranexamic acid for people with mild TBI in the pre-hospital setting, prior to CT scanning, is uncertain. A large randomised controlled trial (CRASH-4) is planned to address this issue, but the economic justification for it has not been established. The aim of the analysis was to estimate the likelihood of tranexamic acid being cost-effective given current evidence, the treatment effects required for cost-effectiveness, and the expected value of performing further research.

METHODS

An early economic decision model compared usual care for mild TBI with and without tranexamic acid, for adults aged 70 and above. The evaluation was performed from a UK healthcare perspective over a lifetime time horizon, with costs reported in 2020 pounds (GBP) and outcomes reported as quality-adjusted life years (QALYs). All analyses used a £20,000 per QALY cost-effectiveness threshold.

RESULTS

In the base case analysis, tranexamic acid was associated with an incremental cost-effectiveness ratio of £4885 per QALY gained, but the likelihood of it being cost-effective was highly dependent on the all-cause mortality treatment effect. The value of perfect information was £22.4 million, and the value of perfect information for parameters that could be collected in a trial was £21.9 million. The all-cause mortality risk ratio for tranexamic acid and the functional outcomes following TBI had the most impact on cost-effectiveness.

CONCLUSIONS

There is a high degree of uncertainty in the cost-effectiveness of tranexamic acid for older adults experiencing mild TBI, meaning there is a high value of performing future research in the UK. The value in a global context is likely to be far higher.

Uptake Rates of Risk-Reducing Surgeries for Women at Increased Risk of Hereditary Breast and Ovarian Cancer Applied to Cost-Effectiveness Analyses: A Scoping Systematic Review

Simple Summary

For women who have tested positive for BRCA mutations, the decision to make use of preventive surgical options, such as risk-reducing mastectomy (RRM) or risk-reducing bilateral salpingo-oophorectomy (RRSO), depends on the women’s personal preferences and the cultural/social context. Among others, the cost-effectiveness of RRM and RRSO can be affected by the uptake rate of these preventive surgical options. Uptake rates of surgery should be given more attention in the conceptualization of health economic modeling studies for RRM and RRSO. Prospective multicenter studies are recommended to reflect regional and national variations in women’s preferences for preventive surgery.

Abstract

The cost-effectiveness of genetic screen-and-treat strategies for women at increased risk for breast and ovarian cancer often depends on the women’s willingness to make use of risk-reducing mastectomy (RRM) or salpingo-oophorectomy (RRSO). To explore the uptake rates of RRM and RRSO applied in health economic modeling studies and the impact of uptake rates on the incremental cost-effectiveness ratios (ICER), we conducted a scoping literature review. In addition, using our own model, we conducted a value of information (VOI) analysis. Among the 19 models included in the review, the uptake rates of RRM ranged from 6% to 47% (RRSO: 10% to 88%). Fifty-seven percent of the models applied retrospective data obtained from registries, hospital records, or questionnaires. According to the models’ deterministic sensitivity analyses, there is a clear trend that a lower uptake rate increased the ICER and vice versa. Our VOI analysis showed high decision uncertainty associated with the uptake rates. In the future, uptake rates should be given more attention in the conceptualization of health economic modeling studies. Prospective studies are recommended to reflect regional and national variations in women’s preferences for preventive surgery.

Simulating Study Data to Support Expected Value of Sample Information Calculations: A Tutorial

The expected value of sample information (EVSI) can be used to prioritize avenues for future research and design studies that support medical decision making and offer value for money spent. EVSI is calculated based on 3 key elements. Two of these, a probabilistic model-based economic evaluation and updating model uncertainty based on simulated data, have been frequently discussed in the literature. By contrast, the third element, simulating data from the proposed studies, has received little attention. This tutorial contributes to bridging this gap by providing a step-by-step guide to simulating study data for EVSI calculations. We discuss a general-purpose algorithm for simulating data and demonstrate its use to simulate 3 different outcome types. We then discuss how to induce correlations in the generated data, how to adjust for common issues in study implementation such as missingness and censoring, and how individual patient data from previous studies can be leveraged to undertake EVSI calculations. For all examples, we provide comprehensive code written in the R language and, where possible, Excel spreadsheets in the supplementary materials. This tutorial facilitates practical EVSI calculations and allows EVSI to be used to prioritize research and design studies.

Population-Based Newborn Screening for Germline TP53 Variants: Clinical Benefits, Cost-Effectiveness, and Value of Further Research

BACKGROUND

Identification of children and infants with Li-Fraumeni syndrome prompts tumor surveillance and allows potential early cancer detection. We assessed the clinical benefits and cost-effectiveness of population-wide newborn screening for TP53 variants (TP53-NBS).

METHODS

We simulated the impact of TP53-NBS using data regarding TP53-associated pediatric cancers and pathogenic or likely pathogenic (P/LP) TP53 variants from Surveillance, Epidemiology, and End Results; ClinVar and gnomAD; and clinical studies. We simulated an annual US birth cohort under usual care and TP53-NBS and estimated clinical benefits, life-years, and costs associated with usual care and TP53-NBS.

RESULTS

Under usual care, of 4 million newborns, 608 (uncertainty interval [UI] = 581-636) individuals would develop TP53-associated cancers before age 20 years. Under TP53-NBS, 894 individuals would have P/LP TP53 variants detected. These individuals would undergo routine surveillance after detection of P/LP TP53 variants decreasing the number of cancer-related deaths by 7.2% (UI = 4.0%-12.1%) overall via early malignancy detection. Compared with usual care, TP53-NBS had an incremental cost-effectiveness ratio of $106 009 per life-year gained. Probabilistic analysis estimated a 40% probability that TP53-NBS would be cost-effective given a $100 000 per life-year gained willingness-to-pay threshold. Using this threshold, a value of information analysis found that additional research on the prevalence of TP53 variants among rhabdomyosarcoma cases would resolve most of the decision uncertainty, resulting in an expected benefit of 349 life-years gained (or $36.6 million).

CONCLUSIONS

We found that TP53-NBS could be cost-effective; however, our findings suggest that further research is needed to reduce the uncertainty in the potential health outcomes and costs associated with TP53-NBS.

Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

BACKGROUND

The expected value of sample information (EVSI) calculates the value of collecting additional information through a research study with a given design. However, standard EVSI analyses do not account for the slow and often incomplete implementation of the treatment recommendations that follow research. Thus, standard EVSI analyses do not correctly capture the value of the study. Previous research has developed measures to calculate the research value while adjusting for implementation challenges, but estimating these measures is a challenge.

METHODS

Based on a method that assumes the implementation level is related to the strength of evidence in favor of the treatment, 2 implementation-adjusted EVSI calculation methods are developed. These novel methods circumvent the need for analytical calculations, which were restricted to settings in which normality could be assumed. The first method developed in this article uses computationally demanding nested simulations, based on the definition of the implementation-adjusted EVSI. The second method is based on adapting the moment matching method, a recently developed efficient EVSI computation method, to adjust for imperfect implementation. The implementation-adjusted EVSI is then calculated with the 2 methods across 3 examples.

RESULTS

The maximum difference between the 2 methods is at most 6% in all examples. The efficient computation method is between 6 and 60 times faster than the nested simulation method in this case study and could be used in practice.

CONCLUSIONS

This article permits the calculation of an implementation-adjusted EVSI using realistic assumptions. The efficient estimation method is accurate and can estimate the implementation-adjusted EVSI in practice. By adapting standard EVSI estimation methods, adjustments for imperfect implementation can be made with the same computational cost as a standard EVSI analysis.

HIGHLIGHTS

Standard expected value of sample information (EVSI) analyses do not account for the fact that treatment implementation following research is often slow and incomplete, meaning they incorrectly capture the value of the study.Two methods, based on nested Monte Carlo sampling and the moment matching EVSI calculation method, are developed to adjust EVSI calculations for imperfect implementation when the speed and level of the implementation of a new treatment depends on the strength of evidence in favor of the treatment.The 2 methods we develop provide similar estimates for the implementation-adjusted EVSI.Our methods extend current EVSI calculation algorithms and thus require limited additional computational complexity.

Expected Value of Sample Information to Guide the Design of Group Sequential Clinical Trials

Introduction

Adaptive designs allow changes to an ongoing trial based on prespecified early examinations of accrued data. Opportunities are potentially being missed to incorporate health economic considerations into the design of these studies.

Methods

We describe how to estimate the expected value of sample information for group sequential design adaptive trials. We operationalize this approach in a hypothetical case study using data from a pilot trial. We report the expected value of sample information and expected net benefit of sampling results for 5 design options for the future full-scale trial including the fixed-sample-size design and the group sequential design using either the Pocock stopping rule or the O’Brien-Fleming stopping rule with 2 or 5 analyses. We considered 2 scenarios relating to 1) using the cost-effectiveness model with a traditional approach to the health economic analysis and 2) adjusting the cost-effectiveness analysis to incorporate the bias-adjusted maximum likelihood estimates of trial outcomes to account for the bias that can be generated in adaptive trials.

Results

The case study demonstrated that the methods developed could be successfully applied in practice. The results showed that the O’Brien-Fleming stopping rule with 2 analyses was the most efficient design with the highest expected net benefit of sampling in the case study.

Conclusions

Cost-effectiveness considerations are unavoidable in budget-constrained, publicly funded health care systems, and adaptive designs can provide an alternative to costly fixed-sample-size designs. We recommend that when planning a clinical trial, expected value of sample information methods be used to compare possible adaptive and nonadaptive trial designs, with appropriate adjustment, to help justify the choice of design characteristics and ensure the cost-effective use of research funding.

Highlights

A guide to value of information methods for prioritising research in health impact modelling

Health impact simulation models are used to predict how a proposed policy or scenario will affect population health outcomes. These models represent the typically-complex systems that describe how the scenarios affect exposures to risk factors for disease or injury (e.g. air pollution or physical inactivity), and how these risk factors are related to measures of population health (e.g. expected survival). These models are informed by multiple sources of data, and are subject to multiple sources of uncertainty. We want to describe which sources of uncertainty contribute most to uncertainty about the estimate or decision arising from the model. Furthermore, we want to decide where further research should be focused to obtain further data to reduce this uncertainty, and what form that research might take. This article presents a tutorial in the use of Value of Information methods for uncertainty analysis and research prioritisation in health impact simulation models. These methods are based on Bayesian decision-theoretic principles, and quantify the expected benefits from further information of different kinds. The expected value of partial perfect information about a parameter measures sensitivity of a decision or estimate to uncertainty about that parameter. The expected value of sample information represents the expected benefit from a specific proposed study to get better information about the parameter. The methods are applicable both to situationswhere the model is used to make a decision between alternative policies, and situations where the model is simply used to estimate a quantity (such as expected gains in survival under a scenario). This paper explains how to calculate and interpret the expected value of information in the context of a simple model describing the health impacts of air pollution from motorised transport. We provide a general-purpose R package and full code to reproduce the example analyses.

Using health technology assessment to set priority, inform target product profiles, and design clinical study for health innovation

Early health technology assessment (early HTA) is a useful tool in guiding the innovation development process in medical technology development. However, the application of early HTA is sub-optimal amongst research and development (R&D) communities due to several challenges. In this paper, we presented a case study of application of early HTA by drawing on the experience from a workshop conducted for the Singapore government's medical technology innovation agency. The framework developed can help maximise the chance of the newly developed technology being accepted and widely used. By providing step-by-step guidance, this work aims to translate early HTA into a practical tool and promote the application of early HTA amongst R&D communities.

The Use of Expert Elicitation among Computational Modeling Studies in Health Research: A Systematic Review

BACKGROUND

Expert elicitation (EE) has been used across disciplines to estimate input parameters for computational modeling research when information is sparse or conflictual.

OBJECTIVES

We conducted a systematic review to compare EE methods used to generate model input parameters in health research.

DATA SOURCES

PubMed and Web of Science.

STUDY ELIGIBILITY

Modeling studies that reported the use of EE as the source for model input probabilities were included if they were published in English before June 2021 and reported health outcomes.

DATA ABSTRACTION AND SYNTHESIS

Studies were classified as "formal" EE methods if they explicitly reported details of their elicitation process. Those that stated use of expert opinion but provided limited information were classified as "indeterminate" methods. In both groups, we abstracted citation details, study design, modeling methodology, a description of elicited parameters, and elicitation methods. Comparisons were made between elicitation methods.

STUDY APPRAISAL

Studies that conducted a formal EE were appraised on the reporting quality of the EE. Quality appraisal was not conducted for studies of indeterminate methods.

RESULTS

The search identified 1520 articles, of which 152 were included. Of the included studies, 40 were classified as formal EE and 112 as indeterminate methods. Most studies were cost-effectiveness analyses (77.6%). Forty-seven indeterminate method studies provided no information on methods for generating estimates. Among formal EEs, the average reporting quality score was 9 out of 16.

LIMITATIONS

Elicitations on nonhealth topics and those reported in the gray literature were not included.

CONCLUSIONS

We found poor reporting of EE methods used in modeling studies, making it difficult to discern meaningful differences in approaches. Improved quality standards for EEs would improve the validity and replicability of computational models.

HIGHLIGHTS

We find extensive use of expert elicitation for the development of model input parameters, but most studies do not provide adequate details of their elicitation methods.Lack of reporting hinders greater discussion of the merits and challenges of using expert elicitation for model input parameter development.There is a need to establish expert elicitation best practices and reporting guidelines.

State of the ART? Two New Tools for Risk Communication in Health Technology Assessments

PURPOSE

Outcomes of health technology assessments (HTA) are uncertain, and decision-making is associated with a risk. This risk, consisting of the probability of making a wrong decision and its impact, is rarely considered in HTA. This hampers transparent and consistent risk assessment and management. The aim of this study was to develop risk communication tools in the context of health technology decision-making under uncertainty.

METHODS

We performed a scoping review of tools for uncertainty and risk communication within HTA using citation pearl-growing. We developed two tools, drawing on existing publications on risk and uncertainty communication for inspiration. Individual semi-structured interviews with HTA stakeholders were performed to identify potential improvements in usefulness, user-friendliness, and information adequacy. Tools were amended and further evaluated in a real-world HTA and workshop with HTA stakeholders.

RESULTS

The identified risk communication tools did not include non-quantified uncertainties, and did not link to risk management strategies. We developed two tools: the Assessment of Risk Table (ART), for a summary of quantified and non-quantified uncertainties and the resulting risk assessment, and the Appraisal of Risk Chart (ARCH), for linking net benefit and risk outcomes to appropriate risk management strategies. Stakeholders appreciated the usefulness of the tools. They also highlighted that more information on local policy options was required for optimal risk management use, and HTA processes may need adapting.

CONCLUSION

The risk communication tools presented here can help assess risk, facilitate communication between analysts and decision-makers, and guide the appropriate use of available risk management strategies.

A Review of Web-Based Tools for Value-of-Information Analysis

Value-of-information analysis (VOI) is a decision-theoretic approach that is used to inform reimbursement decisions, optimise trial design and set research priorities. The application of VOI analysis for informing policy decisions in practice has been limited due, in part, to the perceived complexity associated with the calculation of VOI measures. Recent efforts have resulted in the development of efficient methods to estimate VOI measures and the development of user-friendly web-based tools to facilitate VOI calculations. We review the existing web-based tools including Sheffield Accelerated Value of Information (SAVI), the web interface to the BCEA (Bayesian Cost-Effectiveness Analysis) R package (BCEAweb), Rapid Assessment of Need for Evidence (RANE), and Value of Information for Cardiovascular Trials and Other Comparative Research (VICTOR). We describe what each tool is designed to do, the inputs they require, and the outputs they produce. Finally, we discuss how tools for VOI calculations might be improved in the future to facilitate the use of VOI analysis in practice.

Implementation Barriers to Value of Information Analysis in Health Technology Decision Making: Results From a Process Evaluation

OBJECTIVES

Value of information (VOI) analysis can support health technology assessment decision making, but it is a long way from being standard use. The objective of this study was to understand barriers to the implementation of VOI analysis and propose actions to overcome these.

METHODS

We performed a process evaluation of VOI analysis use within decision making on tomosynthesis versus digital mammography for use in the Dutch breast cancer population screening. Based on steering committee meeting attendance and regular meetings with analysts, we developed a list of barriers to VOI use, which were analyzed using an established diffusion model. We proposed actions to address these barriers. Barriers and actions were discussed and validated in a workshop with stakeholders representing patients, clinicians, regulators, policy advisors, researchers, and the industry.

RESULTS

Consensus was reached on groups of barriers, which included characteristics of VOI analysis itself, stakeholder's attitudes, analysts' and policy makers' skills and knowledge, system readiness, and implementation in the organization. Observed barriers did not only pertain to VOI analysis itself but also to formulating the objective of the assessment, economic modeling, and broader aspects of uncertainty assessment. Actions to overcome these barriers related to organizational changes, knowledge transfer, cultural change, and tools.

CONCLUSIONS

This in-depth analysis of barriers to implementation of VOI analysis and resulting actions and tools may be useful to health technology assessment organizations that wish to implement VOI analysis in technology assessment and research prioritization. Further research should focus on application and evaluation of the proposed actions in real-world assessment processes.

Exploring the Feasibility of Comprehensive Uncertainty Assessment in Health Economic Modeling: A Case Study

OBJECTIVES

Decision makers adopt health technologies based on health economic models that are subject to uncertainty. In an ideal world, these models parameterize all uncertainties and reflect them in the cost-effectiveness probability and risk associated with the adoption. In practice, uncertainty assessment is often incomplete, potentially leading to suboptimal reimbursement recommendations and risk management. This study examines the feasibility of comprehensive uncertainty assessment in health economic models.

METHODS

A state transition model on peripheral arterial disease treatment was used as a case study. Uncertainties were identified and added to the probabilistic sensitivity analysis if possible. Parameter distributions were obtained by expert elicitation, and structural uncertainties were either parameterized or explored in scenario analyses, which were model averaged.

RESULTS

A truly comprehensive uncertainty assessment, parameterizing all uncertainty, could not be achieved. Expert elicitation informed 8 effectiveness, utility, and cost parameters. Uncertainties were parameterized or explored in scenario analyses and with model averaging. Barriers included time and resource constraints, also of clinical experts, and lacking guidance regarding some aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The team's multidisciplinary expertise and existing literature and tools were facilitators.

CONCLUSIONS

While comprehensive uncertainty assessment may not be attainable, improvements in uncertainty assessment in general are no doubt desirable. This requires the development of detailed guidance and hands-on tutorials for methods of uncertainty assessment, in particular aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The issue of benefits of uncertainty assessment versus time and resources needed remains unclear.

Early Cost-effectiveness Analysis of Risk-Based Selection Strategies for Adjuvant Treatment in Stage II Colon Cancer: The Potential Value of Prognostic Molecular Markers

BACKGROUND

To explore the potential value of consensus molecular subtypes (CMS) in stage II colon cancer treatment selection, we carried out an early cost-effectiveness assessment of a CMS-based strategy for adjuvant chemotherapy.

METHODS

We used a Markov cohort model to evaluate three selection strategies: (i) the Dutch guideline strategy (MSS+pT4), (ii) the mutation-based strategy (MSS plus a BRAF and/or KRAS mutation or MSS plus pT4), and (iii) the CMS-based strategy (CMS4 or pT4). Outcomes were number of colon cancer deaths per 1,000 patients, total discounted costs per patient (pp), and quality-adjusted life-years (QALY) pp. The analyses were conducted from a Dutch societal perspective. The robustness of model predictions was assessed in sensitivity analyses. To evaluate the value of future research, we performed a value of information (VOI) analysis.

RESULTS

The Dutch guideline strategy resulted in 8.10 QALYs pp and total costs of €23,660 pp. The CMS-based and mutation-based strategies were more effective and more costly, with 8.12 and 8.13 QALYs pp and €24,643 and €24,542 pp, respectively. Assuming a threshold of €50,000/QALY, the mutation-based strategy was considered as the optimal strategy in an incremental analysis. However, the VOI analysis showed substantial decision uncertainty driven by the molecular markers (expected value of partial perfect information: €18M).

CONCLUSIONS

On the basis of current evidence, our analyses suggest that the mutation-based selection strategy would be the best use of resources. However, the extensive decision uncertainty for the molecular markers does not allow selection of an optimal strategy at present.

IMPACT

Future research is needed to eliminate decision uncertainty driven by molecular markers.

Choosing a Metamodel of a Simulation Model for Uncertainty Quantification

BACKGROUND

Metamodeling may substantially reduce the computational expense of individual-level state transition simulation models (IL-STM) for calibration, uncertainty quantification, and health policy evaluation. However, because of the lack of guidance and readily available computer code, metamodels are still not widely used in health economics and public health. In this study, we provide guidance on how to choose a metamodel for uncertainty quantification.

METHODS

We built a simulation study to evaluate the prediction accuracy and computational expense of metamodels for uncertainty quantification using life-years gained (LYG) by treatment as the IL-STM outcome. We analyzed how metamodel accuracy changes with the characteristics of the simulation model using a linear model (LM), Gaussian process regression (GP), generalized additive models (GAMs), and artificial neural networks (ANNs). Finally, we tested these metamodels in a case study consisting of a probabilistic analysis of a lung cancer IL-STM.

RESULTS

In a scenario with low uncertainty in model parameters (i.e., small confidence interval), sufficient numbers of simulated life histories, and simulation model runs, commonly used metamodels (LM, ANNs, GAMs, and GP) have similar, good accuracy, with errors smaller than 1% for predicting LYG. With a higher level of uncertainty in model parameters, the prediction accuracy of GP and ANN is superior to LM. In the case study, we found that in the worst case, the best metamodel had an error of about 2.1%.

CONCLUSION

To obtain good prediction accuracy, in an efficient way, we recommend starting with LM, and if the resulting accuracy is insufficient, we recommend trying ANNs and eventually also GP regression.