Computing the Expected Value of Sample Information Efficiently: Practical Guidance and Recommendations for Four Model-Based Methods

A Tutorial on Value-Based Adaptive Designs: Could a Value-Based Sequential 2-Arm Design Have Created More Health Economic Value for the Big CACTUS Trial?

OBJECTIVES

Value-based trials aim to maximize the expected net benefit by balancing technology adoption decisions and clinical trial costs. Adaptive trials offer additional efficiency. This article provides guidance on determining whether a value-based sequential design is the best option for an adaptive 2-arm trial, illustrated through a case study.

METHODS

We outlined 4 steps for the value-based sequential approach. The case study re-evaluates the Big CACTUS trial design using pilot trial data and a model-based health economic analysis. Expected net benefit is computed for (1) original fixed design, (2) value-based design with fixed sample size, and (3) optimal value-based sequential design with adaptive stopping. We compare pretrial modeling with the actual Big CACTUS trial results.

RESULTS

Over 10 years, the adoption decision would affect approximately 215 378 patients. Pretrial modeling shows that the expected net benefit minus costs are (1) £102 million for the original fixed design, (2) £107 million (+5.3% higher) for the value-based design with optimal fixed sample size, and (3) £109 million (+6.7% higher) for the optimal value-based sequential design with maximum sample size of 435 per arm. Post hoc analysis using actual Big CACTUS trial data indicates that the value-adaptive trial with a maximum sample size of 95 participant pairs would not have stopped early. Bootstrap simulations reveal a 9.76% probability of early completion with n = 95 pairs compared with 31.50% with n = 435 pairs.

CONCLUSIONS

The 4-step approach to value-based sequential 2-arm design with adaptive stopping was successfully implemented. Further application of value-based adaptive approaches could be useful to assess the efficiency of alternative study designs.

Accurate EVSI Estimation for Nonlinear Models Using the Gaussian Approximation Method

BACKGROUND

The expected value of sample information (EVSI) measures the expected benefits that could be obtained by collecting additional data. Estimating EVSI using the traditional nested Monte Carlo method is computationally expensive, but the recently developed Gaussian approximation (GA) approach can efficiently estimate EVSI across different sample sizes. However, the conventional GA may result in biased EVSI estimates if the decision models are highly nonlinear. This bias may lead to suboptimal study designs when GA is used to optimize the value of different studies. Therefore, we extend the conventional GA approach to improve its performance for nonlinear decision models.

METHODS

Our method provides accurate EVSI estimates by approximating the conditional expectation of the benefit based on 2 steps. First, a Taylor series approximation is applied to estimate the conditional expectation of the benefit as a function of the conditional moments of the parameters of interest using a spline, which is fitted to the samples of the parameters and the corresponding benefits. Next, the conditional moments of parameters are approximated by the conventional GA and Fisher information. The proposed approach is applied to several data collection exercises involving non-Gaussian parameters and nonlinear decision models. Its performance is compared with the nested Monte Carlo method, the conventional GA approach, and the nonparametric regression-based method for EVSI calculation.

RESULTS

The proposed approach provides accurate EVSI estimates across different sample sizes when the parameters of interest are non-Gaussian and the decision models are nonlinear. The computational cost of the proposed method is similar to that of other novel methods.

CONCLUSIONS

The proposed approach can estimate EVSI across sample sizes accurately and efficiently, which may support researchers in determining an economically optimal study design using EVSI.

HIGHLIGHTS

The Gaussian approximation method efficiently estimates the expected value of sample information (EVSI) for clinical trials with varying sample sizes, but it may introduce bias when health economic models have a nonlinear structure.We introduce the spline-based Taylor series approximation method and combine it with the original Gaussian approximation to correct the nonlinearity-induced bias in EVSI estimation.Our approach can provide more precise EVSI estimates for complex decision models without sacrificing computational efficiency, which can enhance the resource allocation strategies from the cost-effective perspective.

Rapid Assessment of the Need for Evidence: Applying the Principles of Value of Information to Research Prioritisation

We propose a short-cut heuristic approach to rapidly estimate value of information (VOI) using information commonly reported in a research funding application to make a case for the need for further evaluative research. We develop a "Rapid VOI" approach, which focuses on uncertainty in the primary outcome of clinical effectiveness and uses this to explore the health consequences of decision uncertainty. We develop a freely accessible online tool, Rapid Assessment of the Need for Evidence (RANE), to allow for the efficient computation of the value of research. As a case study, the method was applied to a proposal for research on shoulder pain rehabilitation. The analysis was included as part of a successful application for research funding to the UK National Institute for Health and Care Research. Our approach enables research funders and applicants to rapidly estimate the value of proposed research. Rapid VOI relies on information that is readily available and reported in research funding applications. Rapid VOI supports research prioritisation and commissioning decisions where there is insufficient time and resources available to develop and validate complex decision-analytic models. The method provides a practical means for implementing VOI in practice, thus providing a starting point for deliberation and contributing to the transparency and accountability of research prioritisation decisions.

Making Drug Approval Decisions in the Face of Uncertainty: Cumulative Evidence versus Value of Information

BACKGROUND

The COVID-19 pandemic underscored the criticality and complexity of decision making for novel treatment approval and further research. Our study aims to assess potential decision-making methodologies, an evaluation vital for refining future public health crisis responses.

METHODS

We compared 4 decision-making approaches to drug approval and research: the Food and Drug Administration's policy decisions, cumulative meta-analysis, a prospective value-of-information (VOI) approach (using information available at the time of decision), and a reference standard (retrospective VOI analysis using information available in hindsight). Possible decisions were to reject, accept, provide emergency use authorization, or allow access to new therapies only in research settings. We used monoclonal antibodies provided to hospitalized COVID-19 patients as a case study, examining the evidence from September 2020 to December 2021 and focusing on each method's capacity to optimize health outcomes and resource allocation.

RESULTS

Our findings indicate a notable discrepancy between policy decisions and the reference standard retrospective VOI approach with expected losses up to $269 billion USD, suggesting suboptimal resource use during the wait for emergency use authorization. Relying solely on cumulative meta-analysis for decision making results in the largest expected loss, while the policy approach showed a loss up to $16 billion and the prospective VOI approach presented the least loss (up to $2 billion).

CONCLUSION

Our research suggests that incorporating VOI analysis may be particularly useful for research prioritization and treatment implementation decisions during pandemics. While the prospective VOI approach was favored in this case study, further studies should validate the ideal decision-making method across various contexts. This study's findings not only enhance our understanding of decision-making strategies during a health crisis but also provide a potential framework for future pandemic responses.

HIGHLIGHTS

This study reviews discrepancies between a reference standard (retrospective VOI, using hindsight information) and 3 conceivable real-time approaches to research-treatment decisions during a pandemic, suggesting suboptimal use of resources.Of all prospective decision-making approaches considered, VOI closely mirrored the reference standard, yielding the least expected value loss across our study timeline.This study illustrates the possible benefit of VOI results and the need for evidence accumulation accompanied by modeling in health technology assessment for emerging therapies.

Thromboprophylaxis during pregnancy and the puerperium: a systematic review and economic evaluation to estimate the value of future research

Background

Pharmacological prophylaxis to prevent venous thromboembolism is currently recommended for women assessed as being at high risk of venous thromboembolism during pregnancy or in the 6 weeks after delivery (the puerperium). The decision to provide thromboprophylaxis involves weighing the benefits, harms and costs, which vary according to the individual's venous thromboembolism risk. It is unclear whether the United Kingdom's current risk stratification approach could be improved by further research.

Objectives

To quantify the current decision uncertainty associated with selecting women who are pregnant or in the puerperium for thromboprophylaxis and to estimate the value of one or more potential future studies that would reduce that uncertainty, while being feasible and acceptable to patients and clinicians.

Methods

A decision-analytic model was developed which was informed by a systematic review of risk assessment models to predict venous thromboembolism in women who are pregnant or in the puerperium. Expected value of perfect information analysis was used to determine which factors are associated with high decision uncertainty and should be the target of future research. To find out whether future studies would be acceptable and feasible, we held workshops with women who have experienced a blood clot or have been offered blood-thinning drugs and surveyed healthcare professionals. Expected value of sample information analysis was used to estimate the value of potential future research studies.

Results

The systematic review included 17 studies, comprising 19 unique externally validated risk assessment models and 1 internally validated model. Estimates of sensitivity and specificity were highly variable ranging from 0% to 100% and 5% to 100%, respectively. Most studies had unclear or high risk of bias and applicability concerns. The decision analysis found that there is substantial decision uncertainty regarding the use of risk assessment models to select high-risk women for antepartum prophylaxis and obese postpartum women for postpartum prophylaxis. The main source of decision uncertainty was uncertainty around the effectiveness of thromboprophylaxis for preventing venous thromboembolism in women who are pregnant or in the puerperium. We found that a randomised controlled trial of thromboprophylaxis in obese postpartum women is likely to have substantial value and is more likely to be acceptable and feasible than a trial recruiting women who have had a previous venous thromboembolism. In unselected postpartum women and women following caesarean section, the poor performance of risk assessment models meant that offering prophylaxis based on these models had less favourable cost effectiveness with lower decision uncertainty.

Limitations

The performance of the risk assessment model for obese postpartum women has not been externally validated.

Conclusions

Future research should focus on estimating the efficacy of pharmacological thromboprophylaxis in pregnancy and the puerperium, and clinical trials would be more acceptable in women who have not had a previous venous thromboembolism.

Study registration

This study is registered as PROSPERO CRD42020221094.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme (NIHR award ref: NIHR131021) and is published in full in Health Technology Assessment; Vol. 28, No. 9. See the NIHR Funding and Awards website for further award information.

Consolidated Health Economic Evaluation Reporting Standards - Value of Information (CHEERS-VOI): Explanation and Elaboration

OBJECTIVES

Although the ISPOR Value of Information (VOI) Task Force's reports outline VOI concepts and provide good-practice recommendations, there is no guidance for reporting VOI analyses. VOI analyses are usually performed alongside economic evaluations for which the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 Statement provides reporting guidelines. Thus, we developed the CHEERS-VOI checklist to provide reporting guidance and checklist to support the transparent, reproducible, and high-quality reporting of VOI analyses.

METHODS

A comprehensive literature review generated a list of 26 candidate reporting items. These candidate items underwent a Delphi procedure with Delphi participants through 3 survey rounds. Participants rated each item on a 9-point Likert scale to indicate its relevance when reporting the minimal, essential information about VOI methods and provided comments. The Delphi results were reviewed at 2-day consensus meetings and the checklist was finalized using anonymous voting.

RESULTS

We had 30, 25, and 24 Delphi respondents in rounds 1, 2, and 3, respectively. After incorporating revisions recommended by the Delphi participants, all 26 candidate items proceeded to the 2-day consensus meetings. The final CHEERS-VOI checklist includes all CHEERS items, but 7 items require elaboration when reporting VOI. Further, 6 new items were added to report information relevant only to VOI (eg, VOI methods applied).

CONCLUSIONS

The CHEERS-VOI checklist should be used when a VOI analysis is performed alongside economic evaluations. The CHEERS-VOI checklist will help decision makers, analysts and peer reviewers in the assessment and interpretation of VOI analyses and thereby increase transparency and rigor in decision making.

General-Purpose Methods for Simulating Survival Data for Expected Value of Sample Information Calculations

BACKGROUND

Expected value of sample information (EVSI) quantifies the expected value to a decision maker of reducing uncertainty by collecting additional data. EVSI calculations require simulating plausible data sets, typically achieved by evaluating quantile functions at random uniform numbers using standard inverse transform sampling (ITS). This is straightforward when closed-form expressions for the quantile function are available, such as for standard parametric survival models, but these are often unavailable when assuming treatment effect waning and for flexible survival models. In these circumstances, the standard ITS method could be implemented by numerically evaluating the quantile functions at each iteration in a probabilistic analysis, but this greatly increases the computational burden. Thus, our study aims to develop general-purpose methods that standardize and reduce the computational burden of the EVSI data-simulation step for survival data.

METHODS

We developed a discrete sampling method and an interpolated ITS method for simulating survival data from a probabilistic sample of survival probabilities over discrete time units. We compared the general-purpose and standard ITS methods using an illustrative partitioned survival model with and without adjustment for treatment effect waning.

RESULTS

The discrete sampling and interpolated ITS methods agree closely with the standard ITS method, with the added benefit of a greatly reduced computational cost in the scenario with adjustment for treatment effect waning.

CONCLUSIONS

We present general-purpose methods for simulating survival data from a probabilistic sample of survival probabilities that greatly reduce the computational burden of the EVSI data-simulation step when we assume treatment effect waning or use flexible survival models. The implementation of our data-simulation methods is identical across all possible survival models and can easily be automated from standard probabilistic decision analyses.

HIGHLIGHTS

Expected value of sample information (EVSI) quantifies the expected value to a decision maker of reducing uncertainty through a given data collection exercise, such as a randomized clinical trial. In this article, we address the problem of computing EVSI when we assume treatment effect waning or use flexible survival models, by developing general-purpose methods that standardize and reduce the computational burden of the EVSI data-generation step for survival data.We developed 2 methods for simulating survival data from a probabilistic sample of survival probabilities over discrete time units, a discrete sampling method and an interpolated inverse transform sampling method, which can be combined with a recently proposed nonparametric EVSI method to accurately estimate EVSI for collecting survival data.Our general-purpose data-simulation methods greatly reduce the computational burden of the EVSI data-simulation step when we assume treatment effect waning or use flexible survival models. The implementation of our data-simulation methods is identical across all possible survival models and can therefore easily be automated from standard probabilistic decision analyses.

Extensions of Health Economic Evaluations in R for Microsoft Excel Users: A Tutorial for Incorporating Heterogeneity and Conducting Value of Information Analyses

Advanced health economic analysis techniques currently performed in Microsoft Excel, such as incorporating heterogeneity, time-dependent transitions and a value of information analysis, can be easily transferred to R. Often the outputs of survival analyses (such as Weibull regression models) will estimate the impacts of correlated patient characteristics on patient outcomes, and are utilised directly as inputs for health economic decision models. This tutorial provides a step-by-step guide of how to conduct such analyses with a Markov model developed in R, and offers a comparison with established analyses performed in Microsoft Excel. This is done through the conversion of a previously published Microsoft Excel case study of a hip replacement surgery cost-effectiveness model. We hope that this paper can act as a facilitator in switching decision models from Microsoft Excel to R for complex health economic analyses, providing open-access code and data, suitable for future adaptation.

Simulation models of sugary drink policies: A scoping review

INTRODUCTION

Simulation modeling methods are an increasingly common tool for projecting the potential health effects of policies to decrease sugar-sweetened beverage (SSB) intake. However, it remains unknown which SSB policies are understudied and how simulation modeling methods could be improved. To inform next steps, we conducted a scoping review to characterize the (1) policies considered and (2) major characteristics of SSB simulation models.

METHODS

We systematically searched 7 electronic databases in 2020, updated in 2021. Two investigators independently screened articles to identify peer-reviewed research using simulation modeling to project the impact of SSB policies on health outcomes. One investigator extracted information about policies considered and key characteristics of models from the full text of included articles. Data were analyzed in 2021-22.

RESULTS

Sixty-one articles were included. Of these, 50 simulated at least one tax policy, most often an ad valorem tax (e.g., 20% tax, n = 25) or volumetric tax (e.g., 1 cent-per-fluid-ounce tax, n = 23). Non-tax policies examined included bans on SSB purchases (n = 5), mandatory reformulation (n = 3), warning labels (n = 2), and portion size policies (n = 2). Policies were typically modeled in populations accounting for age and gender or sex attributes. Most studies focused on weight-related outcomes (n = 54), used cohort, lifetable, or microsimulation modeling methods (n = 34), conducted sensitivity or uncertainty analyses (n = 56), and included supplementary materials (n = 54). Few studies included stakeholders at any point in their process (n = 9) or provided replication code/data (n = 8).

DISCUSSION

Most simulation modeling of SSB policies has focused on tax policies and has been limited in its exploration of heterogenous impacts across population groups. Future research would benefit from refined policy and implementation scenario specifications, thorough assessments of the equity impacts of policies using established methods, and standardized reporting to improve transparency and consistency.

Network Meta-analysis on Disconnected Evidence Networks When Only Aggregate Data Are Available: Modified Methods to Include Disconnected Trials and Single-Arm Studies while Minimizing Bias

BACKGROUND

Network meta-analysis (NMA) requires a connected network of randomized controlled trials (RCTs) and cannot include single-arm studies. Regulators or academics often have only aggregate data. Two aggregate data methods for analyzing disconnected networks are random effects on baseline and aggregate-level matching (ALM). ALM has been used only for single-arm studies, and both methods may bias effect estimates.

METHODS

We modified random effects on baseline to separate RCTs connected to and disconnected from the reference and any single-arm studies, minimizing the introduction of bias. We term our modified method reference prediction. We similarly modified ALM and extended it to include RCTs disconnected from the reference. We tested these methods using constructed data and a simulation study.

RESULTS

In simulations, bias for connected treatments for ALM ranged from -0.0158 to 0.051 and for reference prediction from -0.0107 to 0.083. These were low compared with the true mean effect of 0.5. Coverage ranged from 0.92 to 1.00. In disconnected treatments, bias of ALM ranged from -0.16 to 0.392 and of reference prediction from -0.102 to 0.40, whereas coverage of ALM ranged from 0.30 to 0.82 and of reference prediction from 0.64 to 0.94. Under fixed study effects for disconnected evidence, bias was similar, but coverage was 0.81 to 1.00 for reference prediction and 0.18 to 0.76 for ALM. Trends of similar bias but greater coverage for reference prediction with random study effects were repeated in constructed data.

CONCLUSIONS

Both methods with random study effects seem to minimize bias in treatment connected to the reference. They can estimate treatment effects for disconnected treatments but may be biased. Reference prediction has greater coverage and may be recommended overall.

HIGHLIGHTS

Two methods were modified for network meta-analysis on disconnected networks and for including single-arm observational or interventional studies in network meta-analysis using only aggregate data and for minimizing the bias of effect estimates for treatments only in trials connected to the reference.Reference prediction was developed as a modification of random effects on baseline that keeps analyses of trials connected to the reference separately from those disconnected from the reference and from single-arm studies. The method was further modified to account for correlation in trials with more than 2 arms and, under random study effects, to estimate variance in heterogeneity separately in connected and disconnected evidence.Aggregate-level matching was extended to include trials disconnected from the reference, rather than only single-arm studies. The method was further modified to separately estimate treatment effects and heterogeneity variance in the connected and disconnected evidence and to account for the correlation between arms in trials with more than 2 arms.Performance was assessed using a constructed data example and simulation study.The methods were found to have similar, and sometimes low, bias when estimating the relative effects for disconnected treatments, but reference prediction with random study effects had the greatest coverage.The use of reference prediction with random study effects for disconnected networks is recommended if no individual patient data or alternative real-world evidence is available.

Defining the optimum strategy for identifying adults and children with coeliac disease: systematic review and economic modelling

BACKGROUND

Coeliac disease is an autoimmune disorder triggered by ingesting gluten. It affects approximately 1% of the UK population, but only one in three people is thought to have a diagnosis. Untreated coeliac disease may lead to malnutrition, anaemia, osteoporosis and lymphoma.

OBJECTIVES

The objectives were to define at-risk groups and determine the cost-effectiveness of active case-finding strategies in primary care.

DESIGN

(1) Systematic review of the accuracy of potential diagnostic indicators for coeliac disease. (2) Routine data analysis to develop prediction models for identification of people who may benefit from testing for coeliac disease. (3) Systematic review of the accuracy of diagnostic tests for coeliac disease. (4) Systematic review of the accuracy of genetic tests for coeliac disease (literature search conducted in April 2021). (5) Online survey to identify diagnostic thresholds for testing, starting treatment and referral for biopsy. (6) Economic modelling to identify the cost-effectiveness of different active case-finding strategies, informed by the findings from previous objectives.

DATA SOURCES

For the first systematic review, the following databases were searched from 1997 to April 2021: MEDLINE® (National Library of Medicine, Bethesda, MD, USA), Embase® (Elsevier, Amsterdam, the Netherlands), Cochrane Library, Web of Science™ (Clarivate™, Philadelphia, PA, USA), the World Health Organization International Clinical Trials Registry Platform ( WHO ICTRP ) and the National Institutes of Health Clinical Trials database. For the second systematic review, the following databases were searched from January 1990 to August 2020: MEDLINE, Embase, Cochrane Library, Web of Science, Kleijnen Systematic Reviews ( KSR ) Evidence, WHO ICTRP and the National Institutes of Health Clinical Trials database. For prediction model development, Clinical Practice Research Datalink GOLD, Clinical Practice Research Datalink Aurum and a subcohort of the Avon Longitudinal Study of Parents and Children were used; for estimates for the economic models, Clinical Practice Research Datalink Aurum was used.

REVIEW METHODS

For review 1, cohort and case-control studies reporting on a diagnostic indicator in a population with and a population without coeliac disease were eligible. For review 2, diagnostic cohort studies including patients presenting with coeliac disease symptoms who were tested with serological tests for coeliac disease and underwent a duodenal biopsy as reference standard were eligible. In both reviews, risk of bias was assessed using the quality assessment of diagnostic accuracy studies 2 tool. Bivariate random-effects meta-analyses were fitted, in which binomial likelihoods for the numbers of true positives and true negatives were assumed.

RESULTS

People with dermatitis herpetiformis, a family history of coeliac disease, migraine, anaemia, type 1 diabetes, osteoporosis or chronic liver disease are 1.5-2 times more likely than the general population to have coeliac disease; individual gastrointestinal symptoms were not useful for identifying coeliac disease. For children, women and men, prediction models included 24, 24 and 21 indicators of coeliac disease, respectively. The models showed good discrimination between patients with and patients without coeliac disease, but performed less well when externally validated. Serological tests were found to have good diagnostic accuracy for coeliac disease. Immunoglobulin A tissue transglutaminase had the highest sensitivity and endomysial antibody the highest specificity. There was little improvement when tests were used in combination. Survey respondents (n = 472) wanted to be 66% certain of the diagnosis from a blood test before starting a gluten-free diet if symptomatic, and 90% certain if asymptomatic. Cost-effectiveness analyses found that, among adults, and using serological testing alone, immunoglobulin A tissue transglutaminase was most cost-effective at a 1% pre-test probability (equivalent to population screening). Strategies using immunoglobulin A endomysial antibody plus human leucocyte antigen or human leucocyte antigen plus immunoglobulin A tissue transglutaminase with any pre-test probability had similar cost-effectiveness results, which were also similar to the cost-effectiveness results of immunoglobulin A tissue transglutaminase at a 1% pre-test probability. The most practical alternative for implementation within the NHS is likely to be a combination of human leucocyte antigen and immunoglobulin A tissue transglutaminase testing among those with a pre-test probability above 1.5%. Among children, the most cost-effective strategy was a 10% pre-test probability with human leucocyte antigen plus immunoglobulin A tissue transglutaminase, but there was uncertainty around the most cost-effective pre-test probability. There was substantial uncertainty in economic model results, which means that there would be great value in conducting further research.

LIMITATIONS

The interpretation of meta-analyses was limited by the substantial heterogeneity between the included studies, and most included studies were judged to be at high risk of bias. The main limitations of the prediction models were that we were restricted to diagnostic indicators that were recorded by general practitioners and that, because coeliac disease is underdiagnosed, it is also under-reported in health-care data. The cost-effectiveness model is a simplification of coeliac disease and modelled an average cohort rather than individuals. Evidence was weak on the probability of routine coeliac disease diagnosis, the accuracy of serological and genetic tests and the utility of a gluten-free diet.

CONCLUSIONS

Population screening with immunoglobulin A tissue transglutaminase (1% pre-test probability) and of immunoglobulin A endomysial antibody followed by human leucocyte antigen testing or human leucocyte antigen testing followed by immunoglobulin A tissue transglutaminase with any pre-test probability appear to have similar cost-effectiveness results. As decisions to implement population screening cannot be made based on our economic analysis alone, and given the practical challenges of identifying patients with higher pre-test probabilities, we recommend that human leucocyte antigen combined with immunoglobulin A tissue transglutaminase testing should be considered for adults with at least a 1.5% pre-test probability of coeliac disease, equivalent to having at least one predictor. A more targeted strategy of 10% pre-test probability is recommended for children (e.g. children with anaemia).

FUTURE WORK

Future work should consider whether or not population-based screening for coeliac disease could meet the UK National Screening Committee criteria and whether or not it necessitates a long-term randomised controlled trial of screening strategies. Large prospective cohort studies in which all participants receive accurate tests for coeliac disease are needed.

STUDY REGISTRATION

This study is registered as PROSPERO CRD42019115506 and CRD42020170766.

FUNDING

This project was funded by the National Institute for Health and Care Research ( NIHR ) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 26, No. 44. See the NIHR Journals Library website for further project information.

Emerging Therapies for COVID-19: The Value of Information From More Clinical Trials

OBJECTIVES

The COVID-19 pandemic necessitates time-sensitive policy and implementation decisions regarding new therapies in the face of uncertainty. This study aimed to quantify consequences of approving therapies or pursuing further research: immediate approval, use only in research, approval with research (eg, emergency use authorization), or reject.

METHODS

Using a cohort state-transition model for hospitalized patients with COVID-19, we estimated quality-adjusted life-years (QALYs) and costs associated with the following interventions: hydroxychloroquine, remdesivir, casirivimab-imdevimab, dexamethasone, baricitinib-remdesivir, tocilizumab, lopinavir-ritonavir, interferon beta-1a, and usual care. We used the model outcomes to conduct cost-effectiveness and value of information analyses from a US healthcare perspective and a lifetime horizon.

RESULTS

Assuming a $100 000-per-QALY willingness-to-pay threshold, only remdesivir, casirivimab-imdevimab, dexamethasone, baricitinib-remdesivir, and tocilizumab were (cost-) effective (incremental net health benefit 0.252, 0.164, 0.545, 0.668, and 0.524 QALYs and incremental net monetary benefit $25 249, $16 375, $54 526, $66 826, and $52 378). Our value of information analyses suggest that most value can be obtained if these 5 therapies are approved for immediate use rather than requiring additional randomized controlled trials (RCTs) (net value $20.6 billion, $13.4 billion, $7.4 billion, $54.6 billion, and $7.1 billion), hydroxychloroquine (net value $198 million) is only used in further RCTs if seeking to demonstrate decremental cost-effectiveness and otherwise rejected, and interferon beta-1a and lopinavir-ritonavir are rejected (ie, neither approved nor additional RCTs).

CONCLUSIONS

Estimating the real-time value of collecting additional evidence during the pandemic can inform policy makers and clinicians about the optimal moment to implement therapies and whether to perform further research.

Using decision analysis to support implementation planning in research and practice

BACKGROUND

The process of implementing evidence-based interventions, programs, and policies is difficult and complex. Planning for implementation is critical and likely plays a key role in the long-term impact and sustainability of interventions in practice. However, implementation planning is also difficult. Implementors must choose what to implement and how best to implement it, and each choice has costs and consequences to consider. As a step towards supporting structured and organized implementation planning, we advocate for increased use of decision analysis.

MAIN TEXT

When applied to implementation planning, decision analysis guides users to explicitly define the problem of interest, outline different plans (e.g., interventions/actions, implementation strategies, timelines), and assess the potential outcomes under each alternative in their context. We ground our discussion of decision analysis in the PROACTIVE framework, which guides teams through key steps in decision analyses. This framework includes three phases: (1) definition of the decision problems and overall objectives with purposeful stakeholder engagement, (2) identification and comparison of different alternatives, and (3) synthesis of information on each alternative, incorporating uncertainty. We present three examples to illustrate the breadth of relevant decision analysis approaches to implementation planning.

CONCLUSION

To further the use of decision analysis for implementation planning, we suggest areas for future research and practice: embrace model thinking; build the business case for decision analysis; identify when, how, and for whom decision analysis is more or less useful; improve reporting and transparency of cost data; and increase collaborative opportunities and training.

Methods for Communicating the Impact of Parameter Uncertainty in a Multiple-Strategies Cost-Effectiveness Comparison

Purpose

Analyzing and communicating uncertainty is essential in medical decision making. To judge whether risks are acceptable, policy makers require information on the expected outcomes but also on the uncertainty and potential losses related to the chosen strategy. We aimed to compare methods used to represent the impact of uncertainty in decision problems involving many strategies, enhance existing methods, and provide an open-source and easy-to-use tool.

Methods

We conducted a systematic literature search to identify methods used to represent the impact of uncertainty in cost-effectiveness analyses comparing multiple strategies. We applied the identified methods to probabilistic sensitivity analysis outputs of 3 published decision-analytic models comparing multiple strategies. Subsequently, we compared the following characteristics: type of information conveyed, use of a fixed or flexible willingness-to-pay threshold, output interpretability, and the graphical discriminatory ability. We further proposed adjustments and integration of methods to overcome identified limitations of existing methods.

Results

The literature search resulted in the selection of 9 methods. The 3 methods with the most favorable characteristics to compare many strategies were 1) the cost-effectiveness acceptability curve (CEAC) and cost-effectiveness acceptability frontier (CEAF), 2) the expected loss curve (ELC), and 3) the incremental benefit curve (IBC). The information required to assess confidence in a decision often includes the average loss and the probability of cost-effectiveness associated with each strategy. Therefore, we proposed the integration of information presented in an ELC and CEAC into a single heat map.

Conclusions

This article presents an overview of methods presenting uncertainty in multiple-strategy cost-effectiveness analyses, with their strengths and shortcomings. We proposed a heat map as an alternative method that integrates all relevant information required for health policy and medical decision making.

Highlights

Value of Information Analysis in Models to Inform Health Policy

Value of information (VoI) is a decision-theoretic approach to estimating the expected benefits from collecting further information of different kinds, in scientific problems based on combining one or more sources of data. VoI methods can assess the sensitivity of models to different sources of uncertainty and help to set priorities for further data collection. They have been widely applied in healthcare policy making, but the ideas are general to a range of evidence synthesis and decision problems. This article gives a broad overview of VoI methods, explaining the principles behind them, the range of problems that can be tackled with them, and how they can be implemented, and discusses the ongoing challenges in the area.

Simulating Study Data to Support Expected Value of Sample Information Calculations: A Tutorial

The expected value of sample information (EVSI) can be used to prioritize avenues for future research and design studies that support medical decision making and offer value for money spent. EVSI is calculated based on 3 key elements. Two of these, a probabilistic model-based economic evaluation and updating model uncertainty based on simulated data, have been frequently discussed in the literature. By contrast, the third element, simulating data from the proposed studies, has received little attention. This tutorial contributes to bridging this gap by providing a step-by-step guide to simulating study data for EVSI calculations. We discuss a general-purpose algorithm for simulating data and demonstrate its use to simulate 3 different outcome types. We then discuss how to induce correlations in the generated data, how to adjust for common issues in study implementation such as missingness and censoring, and how individual patient data from previous studies can be leveraged to undertake EVSI calculations. For all examples, we provide comprehensive code written in the R language and, where possible, Excel spreadsheets in the supplementary materials. This tutorial facilitates practical EVSI calculations and allows EVSI to be used to prioritize research and design studies.

Population-Based Newborn Screening for Germline TP53 Variants: Clinical Benefits, Cost-Effectiveness, and Value of Further Research

BACKGROUND

Identification of children and infants with Li-Fraumeni syndrome prompts tumor surveillance and allows potential early cancer detection. We assessed the clinical benefits and cost-effectiveness of population-wide newborn screening for TP53 variants (TP53-NBS).

METHODS

We simulated the impact of TP53-NBS using data regarding TP53-associated pediatric cancers and pathogenic or likely pathogenic (P/LP) TP53 variants from Surveillance, Epidemiology, and End Results; ClinVar and gnomAD; and clinical studies. We simulated an annual US birth cohort under usual care and TP53-NBS and estimated clinical benefits, life-years, and costs associated with usual care and TP53-NBS.

RESULTS

Under usual care, of 4 million newborns, 608 (uncertainty interval [UI] = 581-636) individuals would develop TP53-associated cancers before age 20 years. Under TP53-NBS, 894 individuals would have P/LP TP53 variants detected. These individuals would undergo routine surveillance after detection of P/LP TP53 variants decreasing the number of cancer-related deaths by 7.2% (UI = 4.0%-12.1%) overall via early malignancy detection. Compared with usual care, TP53-NBS had an incremental cost-effectiveness ratio of $106 009 per life-year gained. Probabilistic analysis estimated a 40% probability that TP53-NBS would be cost-effective given a $100 000 per life-year gained willingness-to-pay threshold. Using this threshold, a value of information analysis found that additional research on the prevalence of TP53 variants among rhabdomyosarcoma cases would resolve most of the decision uncertainty, resulting in an expected benefit of 349 life-years gained (or $36.6 million).

CONCLUSIONS

We found that TP53-NBS could be cost-effective; however, our findings suggest that further research is needed to reduce the uncertainty in the potential health outcomes and costs associated with TP53-NBS.

Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

BACKGROUND

The expected value of sample information (EVSI) calculates the value of collecting additional information through a research study with a given design. However, standard EVSI analyses do not account for the slow and often incomplete implementation of the treatment recommendations that follow research. Thus, standard EVSI analyses do not correctly capture the value of the study. Previous research has developed measures to calculate the research value while adjusting for implementation challenges, but estimating these measures is a challenge.

METHODS

Based on a method that assumes the implementation level is related to the strength of evidence in favor of the treatment, 2 implementation-adjusted EVSI calculation methods are developed. These novel methods circumvent the need for analytical calculations, which were restricted to settings in which normality could be assumed. The first method developed in this article uses computationally demanding nested simulations, based on the definition of the implementation-adjusted EVSI. The second method is based on adapting the moment matching method, a recently developed efficient EVSI computation method, to adjust for imperfect implementation. The implementation-adjusted EVSI is then calculated with the 2 methods across 3 examples.

RESULTS

The maximum difference between the 2 methods is at most 6% in all examples. The efficient computation method is between 6 and 60 times faster than the nested simulation method in this case study and could be used in practice.

CONCLUSIONS

This article permits the calculation of an implementation-adjusted EVSI using realistic assumptions. The efficient estimation method is accurate and can estimate the implementation-adjusted EVSI in practice. By adapting standard EVSI estimation methods, adjustments for imperfect implementation can be made with the same computational cost as a standard EVSI analysis.

HIGHLIGHTS

Standard expected value of sample information (EVSI) analyses do not account for the fact that treatment implementation following research is often slow and incomplete, meaning they incorrectly capture the value of the study.Two methods, based on nested Monte Carlo sampling and the moment matching EVSI calculation method, are developed to adjust EVSI calculations for imperfect implementation when the speed and level of the implementation of a new treatment depends on the strength of evidence in favor of the treatment.The 2 methods we develop provide similar estimates for the implementation-adjusted EVSI.Our methods extend current EVSI calculation algorithms and thus require limited additional computational complexity.

Expected Value of Sample Information to Guide the Design of Group Sequential Clinical Trials

Introduction

Adaptive designs allow changes to an ongoing trial based on prespecified early examinations of accrued data. Opportunities are potentially being missed to incorporate health economic considerations into the design of these studies.

Methods

We describe how to estimate the expected value of sample information for group sequential design adaptive trials. We operationalize this approach in a hypothetical case study using data from a pilot trial. We report the expected value of sample information and expected net benefit of sampling results for 5 design options for the future full-scale trial including the fixed-sample-size design and the group sequential design using either the Pocock stopping rule or the O’Brien-Fleming stopping rule with 2 or 5 analyses. We considered 2 scenarios relating to 1) using the cost-effectiveness model with a traditional approach to the health economic analysis and 2) adjusting the cost-effectiveness analysis to incorporate the bias-adjusted maximum likelihood estimates of trial outcomes to account for the bias that can be generated in adaptive trials.

Results

The case study demonstrated that the methods developed could be successfully applied in practice. The results showed that the O’Brien-Fleming stopping rule with 2 analyses was the most efficient design with the highest expected net benefit of sampling in the case study.

Conclusions

Cost-effectiveness considerations are unavoidable in budget-constrained, publicly funded health care systems, and adaptive designs can provide an alternative to costly fixed-sample-size designs. We recommend that when planning a clinical trial, expected value of sample information methods be used to compare possible adaptive and nonadaptive trial designs, with appropriate adjustment, to help justify the choice of design characteristics and ensure the cost-effective use of research funding.

Highlights

Economic Evaluation of Transperineal versus Transrectal Devices for Local Anaesthetic Prostate Biopsies

BACKGROUND

Biopsy of the prostate for suspected cancer is usually performed transrectally under local anaesthesia in the outpatient clinic setting. As this involves piercing the bowel wall, the procedure is associated with a risk of infection. Recently, devices that facilitate transperineal biopsy approaches have been developed that avoid piercing the bowel and so should reduce the risk of infection.

OBJECTIVE

The aim of this study was to estimate the cost effectiveness of transperineal versus transrectal ultrasound-guided local anaesthesia procedures for prostate biopsy from the perspective of the UK NHS and to estimate the value of further research in the area.

METHODS

a) Decision tree and Markov model synthesising all relevant evidence estimating the life-time costs and QALYs accrued from each biopsy mode. b) Value of information analysis to predict the return from further research and thus guide future research efforts.

RESULTS

Transperineal biopsy yields an ICER below £20,000 per QALY gained at a per-procedure device acquisition cost below £81, or £41 for cost-neutrality. These results are driven by differences in consumables cost, reduced cost of treating infections, and QALY gains associated with reduced infections. There is value in future research on the diagnostic accuracy of transperineal versus transrectal biopsies and the incidence of iatrogenic infection and sepsis; consideration should be given to enriching the patient population with men with intermediate-risk disease.

CONCLUSIONS

Transperineal biopsy devices may be cost effective compared with transrectal biopsy at per-procedure acquisition costs below £81 and cost-neutral if under £41. Future research is required to confirm or refute these findings, particularly randomised comparisons of the diagnostic accuracy and infection risks between the methods.

A guide to value of information methods for prioritising research in health impact modelling

Health impact simulation models are used to predict how a proposed policy or scenario will affect population health outcomes. These models represent the typically-complex systems that describe how the scenarios affect exposures to risk factors for disease or injury (e.g. air pollution or physical inactivity), and how these risk factors are related to measures of population health (e.g. expected survival). These models are informed by multiple sources of data, and are subject to multiple sources of uncertainty. We want to describe which sources of uncertainty contribute most to uncertainty about the estimate or decision arising from the model. Furthermore, we want to decide where further research should be focused to obtain further data to reduce this uncertainty, and what form that research might take. This article presents a tutorial in the use of Value of Information methods for uncertainty analysis and research prioritisation in health impact simulation models. These methods are based on Bayesian decision-theoretic principles, and quantify the expected benefits from further information of different kinds. The expected value of partial perfect information about a parameter measures sensitivity of a decision or estimate to uncertainty about that parameter. The expected value of sample information represents the expected benefit from a specific proposed study to get better information about the parameter. The methods are applicable both to situationswhere the model is used to make a decision between alternative policies, and situations where the model is simply used to estimate a quantity (such as expected gains in survival under a scenario). This paper explains how to calculate and interpret the expected value of information in the context of a simple model describing the health impacts of air pollution from motorised transport. We provide a general-purpose R package and full code to reproduce the example analyses.

Much ado about something: a response to "COVID-19: underpowered randomised trials, or no randomised trials?"

Non-pharmaceutical interventions (NPI) for infectious diseases such as COVID-19 are particularly challenging given the complexities of what is both practical and ethical to randomize. We are often faced with the difficult decision between having weak trials or not having a trial at all. In a recent article, Dr. Atle Fretheim argues that statistically underpowered studies are still valuable, particularly in conjunction with other similar studies in meta-analysis in the context of the DANMASK-19 trial, asking "Surely, some trial evidence must be better than no trial evidence?" However, informative trials are not always feasible, and feasible trials are not always informative. In some cases, even a well-conducted but weakly designed and/or underpowered trial such as DANMASK-19 may be uninformative or worse, both individually and in a body of literature. Meta-analysis, for example, can only resolve issues of statistical power if there is a reasonable expectation of compatible well-designed trials. Uninformative designs may also invite misinformation. Here, we make the case that-when considering informativeness, ethics, and opportunity costs in addition to statistical power-"nothing" is often the better choice.

A Review of Web-Based Tools for Value-of-Information Analysis

Value-of-information analysis (VOI) is a decision-theoretic approach that is used to inform reimbursement decisions, optimise trial design and set research priorities. The application of VOI analysis for informing policy decisions in practice has been limited due, in part, to the perceived complexity associated with the calculation of VOI measures. Recent efforts have resulted in the development of efficient methods to estimate VOI measures and the development of user-friendly web-based tools to facilitate VOI calculations. We review the existing web-based tools including Sheffield Accelerated Value of Information (SAVI), the web interface to the BCEA (Bayesian Cost-Effectiveness Analysis) R package (BCEAweb), Rapid Assessment of Need for Evidence (RANE), and Value of Information for Cardiovascular Trials and Other Comparative Research (VICTOR). We describe what each tool is designed to do, the inputs they require, and the outputs they produce. Finally, we discuss how tools for VOI calculations might be improved in the future to facilitate the use of VOI analysis in practice.

Implementation Barriers to Value of Information Analysis in Health Technology Decision Making: Results From a Process Evaluation

OBJECTIVES

Value of information (VOI) analysis can support health technology assessment decision making, but it is a long way from being standard use. The objective of this study was to understand barriers to the implementation of VOI analysis and propose actions to overcome these.

METHODS

We performed a process evaluation of VOI analysis use within decision making on tomosynthesis versus digital mammography for use in the Dutch breast cancer population screening. Based on steering committee meeting attendance and regular meetings with analysts, we developed a list of barriers to VOI use, which were analyzed using an established diffusion model. We proposed actions to address these barriers. Barriers and actions were discussed and validated in a workshop with stakeholders representing patients, clinicians, regulators, policy advisors, researchers, and the industry.

RESULTS

Consensus was reached on groups of barriers, which included characteristics of VOI analysis itself, stakeholder's attitudes, analysts' and policy makers' skills and knowledge, system readiness, and implementation in the organization. Observed barriers did not only pertain to VOI analysis itself but also to formulating the objective of the assessment, economic modeling, and broader aspects of uncertainty assessment. Actions to overcome these barriers related to organizational changes, knowledge transfer, cultural change, and tools.

CONCLUSIONS

This in-depth analysis of barriers to implementation of VOI analysis and resulting actions and tools may be useful to health technology assessment organizations that wish to implement VOI analysis in technology assessment and research prioritization. Further research should focus on application and evaluation of the proposed actions in real-world assessment processes.

Multilevel and Quasi Monte Carlo Methods for the Calculation of the Expected Value of Partial Perfect Information

The expected value of partial perfect information (EVPPI) provides an upper bound on the value of collecting further evidence on a set of inputs to a cost-effectiveness decision model. Standard Monte Carlo estimation of EVPPI is computationally expensive as it requires nested simulation. Alternatives based on regression approximations to the model have been developed but are not practicable when the number of uncertain parameters of interest is large and when parameter estimates are highly correlated. The error associated with the regression approximation is difficult to determine, while MC allows the bias and precision to be controlled. In this article, we explore the potential of quasi Monte Carlo (QMC) and multilevel Monte Carlo (MLMC) estimation to reduce the computational cost of estimating EVPPI by reducing the variance compared with MC while preserving accuracy. We also develop methods to apply QMC and MLMC to EVPPI, addressing particular challenges that arise where Markov chain Monte Carlo (MCMC) has been used to estimate input parameter distributions. We illustrate the methods using 2 examples: a simplified decision tree model for treatments for depression and a complex Markov model for treatments to prevent stroke in atrial fibrillation, both of which use MCMC inputs. We compare the performance of QMC and MLMC with MC and the approximation techniques of generalized additive model (GAM) regression, Gaussian process (GP) regression, and integrated nested Laplace approximations (INLA-GP). We found QMC and MLMC to offer substantial computational savings when parameter sets are large and correlated and when the EVPPI is large. We also found that GP and INLA-GP were biased in those situations, whereas GAM cannot estimate EVPPI for large parameter sets.

Exploring the Feasibility of Comprehensive Uncertainty Assessment in Health Economic Modeling: A Case Study

OBJECTIVES

Decision makers adopt health technologies based on health economic models that are subject to uncertainty. In an ideal world, these models parameterize all uncertainties and reflect them in the cost-effectiveness probability and risk associated with the adoption. In practice, uncertainty assessment is often incomplete, potentially leading to suboptimal reimbursement recommendations and risk management. This study examines the feasibility of comprehensive uncertainty assessment in health economic models.

METHODS

A state transition model on peripheral arterial disease treatment was used as a case study. Uncertainties were identified and added to the probabilistic sensitivity analysis if possible. Parameter distributions were obtained by expert elicitation, and structural uncertainties were either parameterized or explored in scenario analyses, which were model averaged.

RESULTS

A truly comprehensive uncertainty assessment, parameterizing all uncertainty, could not be achieved. Expert elicitation informed 8 effectiveness, utility, and cost parameters. Uncertainties were parameterized or explored in scenario analyses and with model averaging. Barriers included time and resource constraints, also of clinical experts, and lacking guidance regarding some aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The team's multidisciplinary expertise and existing literature and tools were facilitators.

CONCLUSIONS

While comprehensive uncertainty assessment may not be attainable, improvements in uncertainty assessment in general are no doubt desirable. This requires the development of detailed guidance and hands-on tutorials for methods of uncertainty assessment, in particular aspects of expert elicitation, evidence aggregation, and handling of structural uncertainty. The issue of benefits of uncertainty assessment versus time and resources needed remains unclear.

Prioritisation and design of clinical trials

Clinical trials require participation of numerous patients, enormous research resources and substantial public funding. Time-consuming trials lead to delayed implementation of beneficial interventions and to reduced benefit to patients. This manuscript discusses two methods for the allocation of research resources and reviews a framework for prioritisation and design of clinical trials. The traditional error-driven approach of clinical trial design controls for type I and II errors. However, controlling for those statistical errors has limited relevance to policy makers. Therefore, this error-driven approach can be inefficient, waste research resources and lead to research with limited impact on daily practice. The novel value-driven approach assesses the currently available evidence and focuses on designing clinical trials that directly inform policy and treatment decisions. Estimating the net value of collecting further information, prior to undertaking a trial, informs a decision maker whether a clinical or health policy decision can be made with current information or if collection of extra evidence is justified. Additionally, estimating the net value of new information guides study design, data collection choices, and sample size estimation. The value-driven approach ensures the efficient use of research resources, reduces unnecessary burden to trial participants, and accelerates implementation of beneficial healthcare interventions.

Building a trusted framework for uncertainty assessment in rare diseases: suggestions for improvement (Response to "TRUST4RD: tool for reducing uncertainties in the evidence generation for specialised treatments for rare diseases")

The aim of this letter to the editor is to provide a comprehensive summary of uncertainty assessment in Health Technology Assessment, with a focus on transferability to the setting of rare diseases. The authors of "TRUST4RD: tool for reducing uncertainties in the evidence generation for specialised treatments for rare diseases" presented recommendations for reducing uncertainty in rare diseases. Their article is of great importance but unfortunately suffers from a lack of references to the wider uncertainty in Health Technology Assessment and research prioritisation literature and consequently fails to provide a trusted framework for decision-making in rare diseases. In this letter to the editor we critique the authors' tool and provide pointers as to how their proposal can be strengthened. We present references to the literature, including our own tool for uncertainty assessment (TRUST; unrelated to the authors' research), apply TRUST to two assessments of orphan drugs in rare diseases and provide a broader perspective on uncertainty and risk management in rare diseases, including a detailed research agenda.

How can health economics be used in the design and analysis of adaptive clinical trials? A qualitative analysis

INTRODUCTION

Adaptive designs offer a flexible approach, allowing changes to a trial based on examinations of the data as it progresses. Adaptive clinical trials are becoming a popular choice, as the prudent use of finite research budgets and accurate decision-making are priorities for healthcare providers around the world. The methods of health economics, which aim to maximise the health gained for money spent, could be incorporated into the design and analysis of adaptive clinical trials to make them more efficient. We aimed to understand the perspectives of stakeholders in health technology assessments to inform recommendations for the use of health economics in adaptive clinical trials.

METHODS

A qualitative study explored the attitudes of key stakeholders-including researchers, decision-makers and members of the public-towards the use of health economics in the design and analysis of adaptive clinical trials. Data were collected using interviews and focus groups (29 participants). A framework analysis was used to identify themes in the transcripts.

RESULTS

It was considered that answering the clinical research question should be the priority in a clinical trial, notwithstanding the importance of cost-effectiveness for decision-making. Concerns raised by participants included handling the volatile nature of cost data at interim analyses; implementing this approach in global trials; resourcing adaptive trials which are designed and adapted based on health economic outcomes; and training stakeholders in these methods so that they can be implemented and appropriately interpreted.

CONCLUSION

The use of health economics in the design and analysis of adaptive clinical trials has the potential to increase the efficiency of health technology assessments worldwide. Recommendations are made concerning the development of methods allowing the use of health economics in adaptive clinical trials, and suggestions are given to facilitate their implementation in practice.