Estimating the Expected Value of Sample Information across Different Sample Sizes Using Moment Matching and Nonlinear Regression

Consolidated Health Economic Evaluation Reporting Standards - Value of Information (CHEERS-VOI): Explanation and Elaboration

OBJECTIVES

Although the ISPOR Value of Information (VOI) Task Force's reports outline VOI concepts and provide good-practice recommendations, there is no guidance for reporting VOI analyses. VOI analyses are usually performed alongside economic evaluations for which the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 Statement provides reporting guidelines. Thus, we developed the CHEERS-VOI checklist to provide reporting guidance and checklist to support the transparent, reproducible, and high-quality reporting of VOI analyses.

METHODS

A comprehensive literature review generated a list of 26 candidate reporting items. These candidate items underwent a Delphi procedure with Delphi participants through 3 survey rounds. Participants rated each item on a 9-point Likert scale to indicate its relevance when reporting the minimal, essential information about VOI methods and provided comments. The Delphi results were reviewed at 2-day consensus meetings and the checklist was finalized using anonymous voting.

RESULTS

We had 30, 25, and 24 Delphi respondents in rounds 1, 2, and 3, respectively. After incorporating revisions recommended by the Delphi participants, all 26 candidate items proceeded to the 2-day consensus meetings. The final CHEERS-VOI checklist includes all CHEERS items, but 7 items require elaboration when reporting VOI. Further, 6 new items were added to report information relevant only to VOI (eg, VOI methods applied).

CONCLUSIONS

The CHEERS-VOI checklist should be used when a VOI analysis is performed alongside economic evaluations. The CHEERS-VOI checklist will help decision makers, analysts and peer reviewers in the assessment and interpretation of VOI analyses and thereby increase transparency and rigor in decision making.

Value of Information Analysis in Models to Inform Health Policy

Value of information (VoI) is a decision-theoretic approach to estimating the expected benefits from collecting further information of different kinds, in scientific problems based on combining one or more sources of data. VoI methods can assess the sensitivity of models to different sources of uncertainty and help to set priorities for further data collection. They have been widely applied in healthcare policy making, but the ideas are general to a range of evidence synthesis and decision problems. This article gives a broad overview of VoI methods, explaining the principles behind them, the range of problems that can be tackled with them, and how they can be implemented, and discusses the ongoing challenges in the area.

Simulating Study Data to Support Expected Value of Sample Information Calculations: A Tutorial

The expected value of sample information (EVSI) can be used to prioritize avenues for future research and design studies that support medical decision making and offer value for money spent. EVSI is calculated based on 3 key elements. Two of these, a probabilistic model-based economic evaluation and updating model uncertainty based on simulated data, have been frequently discussed in the literature. By contrast, the third element, simulating data from the proposed studies, has received little attention. This tutorial contributes to bridging this gap by providing a step-by-step guide to simulating study data for EVSI calculations. We discuss a general-purpose algorithm for simulating data and demonstrate its use to simulate 3 different outcome types. We then discuss how to induce correlations in the generated data, how to adjust for common issues in study implementation such as missingness and censoring, and how individual patient data from previous studies can be leveraged to undertake EVSI calculations. For all examples, we provide comprehensive code written in the R language and, where possible, Excel spreadsheets in the supplementary materials. This tutorial facilitates practical EVSI calculations and allows EVSI to be used to prioritize research and design studies.

Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

BACKGROUND

The expected value of sample information (EVSI) calculates the value of collecting additional information through a research study with a given design. However, standard EVSI analyses do not account for the slow and often incomplete implementation of the treatment recommendations that follow research. Thus, standard EVSI analyses do not correctly capture the value of the study. Previous research has developed measures to calculate the research value while adjusting for implementation challenges, but estimating these measures is a challenge.

METHODS

Based on a method that assumes the implementation level is related to the strength of evidence in favor of the treatment, 2 implementation-adjusted EVSI calculation methods are developed. These novel methods circumvent the need for analytical calculations, which were restricted to settings in which normality could be assumed. The first method developed in this article uses computationally demanding nested simulations, based on the definition of the implementation-adjusted EVSI. The second method is based on adapting the moment matching method, a recently developed efficient EVSI computation method, to adjust for imperfect implementation. The implementation-adjusted EVSI is then calculated with the 2 methods across 3 examples.

RESULTS

The maximum difference between the 2 methods is at most 6% in all examples. The efficient computation method is between 6 and 60 times faster than the nested simulation method in this case study and could be used in practice.

CONCLUSIONS

This article permits the calculation of an implementation-adjusted EVSI using realistic assumptions. The efficient estimation method is accurate and can estimate the implementation-adjusted EVSI in practice. By adapting standard EVSI estimation methods, adjustments for imperfect implementation can be made with the same computational cost as a standard EVSI analysis.

HIGHLIGHTS

Standard expected value of sample information (EVSI) analyses do not account for the fact that treatment implementation following research is often slow and incomplete, meaning they incorrectly capture the value of the study.Two methods, based on nested Monte Carlo sampling and the moment matching EVSI calculation method, are developed to adjust EVSI calculations for imperfect implementation when the speed and level of the implementation of a new treatment depends on the strength of evidence in favor of the treatment.The 2 methods we develop provide similar estimates for the implementation-adjusted EVSI.Our methods extend current EVSI calculation algorithms and thus require limited additional computational complexity.

Prioritisation and design of clinical trials

Clinical trials require participation of numerous patients, enormous research resources and substantial public funding. Time-consuming trials lead to delayed implementation of beneficial interventions and to reduced benefit to patients. This manuscript discusses two methods for the allocation of research resources and reviews a framework for prioritisation and design of clinical trials. The traditional error-driven approach of clinical trial design controls for type I and II errors. However, controlling for those statistical errors has limited relevance to policy makers. Therefore, this error-driven approach can be inefficient, waste research resources and lead to research with limited impact on daily practice. The novel value-driven approach assesses the currently available evidence and focuses on designing clinical trials that directly inform policy and treatment decisions. Estimating the net value of collecting further information, prior to undertaking a trial, informs a decision maker whether a clinical or health policy decision can be made with current information or if collection of extra evidence is justified. Additionally, estimating the net value of new information guides study design, data collection choices, and sample size estimation. The value-driven approach ensures the efficient use of research resources, reduces unnecessary burden to trial participants, and accelerates implementation of beneficial healthcare interventions.

Optimal Allocation of Research Funds under a Budget Constraint

Purpose. Health economic evaluations that include the expected value of sample information support implementation decisions as well as decisions about further research. However, just as decision makers must consider portfolios of implementation spending, they must also identify the optimal portfolio of research investments. Methods. Under a fixed research budget, a decision maker determines which studies to fund; additional budget allocated to one study to increase the study sample size implies less budget available to collect information to reduce decision uncertainty in other implementation decisions. We employ a budget-constrained portfolio optimization framework in which the decisions are whether to invest in a study and at what sample size. The objective is to maximize the sum of the studies' population expected net benefit of sampling (ENBS). We show how to determine the optimal research portfolio and study-specific levels of investment. We demonstrate our framework with a stylized example to illustrate solution features and a real-world application using 6 published cost-effectiveness analyses. Results. Among the studies selected for nonzero investment, the optimal sample size occurs at the point at which the marginal population ENBS divided by the marginal cost of additional sampling is the same for all studies. Compared with standard ENBS optimization without a research budget constraint, optimal budget-constrained sample sizes are typically smaller but allow more studies to be funded. Conclusions. The budget constraint for research studies directly implies that the optimal sample size for additional research is not the point at which the ENBS is maximized for individual studies. A portfolio optimization approach can yield higher total ENBS. Ultimately, there is a maximum willingness to pay for incremental information that determines optimal sample sizes.

Computing the Expected Value of Sample Information Efficiently: Practical Guidance and Recommendations for Four Model-Based Methods

Value of information (VOI) analyses can help policy makers make informed decisions about whether to conduct and how to design future studies. Historically a computationally expensive method to compute the expected value of sample information (EVSI) restricted the use of VOI to simple decision models and study designs. Recently, 4 EVSI approximation methods have made such analyses more feasible and accessible. Members of the Collaborative Network for Value of Information (ConVOI) compared the inputs, the analyst's expertise and skills, and the software required for the 4 recently developed EVSI approximation methods. Our report provides practical guidance and recommendations to help inform the choice between the 4 efficient EVSI estimation methods. More specifically, this report provides: (1) a step-by-step guide to the methods' use, (2) the expertise and skills required to implement the methods, and (3) method recommendations based on the features of decision-analytic problems.