Implementing Generalized Additive Models to Estimate the Expected Value of Sample Information in a Microsimulation Model: Results of Three Case Studies

Consolidated Health Economic Evaluation Reporting Standards - Value of Information (CHEERS-VOI): Explanation and Elaboration

OBJECTIVES

Although the ISPOR Value of Information (VOI) Task Force's reports outline VOI concepts and provide good-practice recommendations, there is no guidance for reporting VOI analyses. VOI analyses are usually performed alongside economic evaluations for which the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 Statement provides reporting guidelines. Thus, we developed the CHEERS-VOI checklist to provide reporting guidance and checklist to support the transparent, reproducible, and high-quality reporting of VOI analyses.

METHODS

A comprehensive literature review generated a list of 26 candidate reporting items. These candidate items underwent a Delphi procedure with Delphi participants through 3 survey rounds. Participants rated each item on a 9-point Likert scale to indicate its relevance when reporting the minimal, essential information about VOI methods and provided comments. The Delphi results were reviewed at 2-day consensus meetings and the checklist was finalized using anonymous voting.

RESULTS

We had 30, 25, and 24 Delphi respondents in rounds 1, 2, and 3, respectively. After incorporating revisions recommended by the Delphi participants, all 26 candidate items proceeded to the 2-day consensus meetings. The final CHEERS-VOI checklist includes all CHEERS items, but 7 items require elaboration when reporting VOI. Further, 6 new items were added to report information relevant only to VOI (eg, VOI methods applied).

CONCLUSIONS

The CHEERS-VOI checklist should be used when a VOI analysis is performed alongside economic evaluations. The CHEERS-VOI checklist will help decision makers, analysts and peer reviewers in the assessment and interpretation of VOI analyses and thereby increase transparency and rigor in decision making.

The Value-of-Information and Value-of-Implementation from Clinical Trials of Diagnostic Tests for HIV-Associated Tuberculosis: A Modeling Analysis

Objectives. Conventional value-of-information (VOI) analysis assumes complete uptake of an optimal decision. We employed an extended framework that includes value-of-implementation (VOM)-the benefit of encouraging adoption of an optimal strategy-and estimated how future trials of diagnostic tests for HIV-associated tuberculosis could improve public health decision making and clinical and economic outcomes. Methods. We evaluated the clinical outcomes and costs, given current information, of 3 tuberculosis screening strategies among hospitalized people with HIV in South Africa: sputum Xpert (Xpert), sputum Xpert plus urine AlereLAM (Xpert+AlereLAM), and sputum Xpert plus the newer, more sensitive, and costlier urine FujiLAM (Xpert+FujiLAM). We projected the incremental net monetary benefit (INMB) of decision making based on results of a trial comparing mortality with each strategy, rather than decision making based solely on current knowledge of FujiLAM's improved diagnostic performance. We used a validated microsimulation to estimate VOI (the INMB of reducing parameter uncertainty before decision making) and VOM (the INMB of encouraging adoption of an optimal strategy). Results. With current information, adopting Xpert+FujiLAM yields 0.4 additional life-years/person compared with current practices (assumed 50% Xpert and 50% Xpert+AlereLAM). While the decision to adopt this optimal strategy is unaffected by information from the clinical trial (VOI = $ 0 at $3,000/year-of-life saved willingness-to-pay threshold), there is value in scaling up implementation of Xpert+FujiLAM, which results in an INMB (representing VOM) of $650 million over 5 y. Conclusions. Conventional VOI methods account for the value of switching to a new optimal strategy based on trial data but fail to account for the persuasive value of trials in increasing uptake of the optimal strategy. Evaluation of trials should include a focus on their value in reducing barriers to implementation.

Highlights

In conventional VOI analysis, it is assumed that the optimal decision will always be adopted even without a trial. This can potentially lead to an underestimation of the value of trials when adoption requires new clinical trial evidence. To capture the influence that a trial may have on decision makers' willingness to adopt the optimal decision, we also consider value-of-implementation (VOM), a metric quantifying the benefit of new study information in promoting wider adoption of the optimal strategy. The overall value-of-a-trial (VOT) includes both VOI and VOM.Our model-based analysis suggests that the information obtained from a trial of screening strategies for HIV-associated tuberculosis in South Africa would have no value, when measured using traditional methods of VOI assessment. A novel strategy, which includes the urine FujiLAM test, is optimal from a health economic standpoint but is underutilized. A trial would reduce uncertainties around downstream health outcomes but likely would not change the optimal decision. The high VOT (nearly $700 million over 5 y) lies solely in promoting uptake of FujiLAM, represented as VOM.Our results highlight the importance of employing a more comprehensive approach for evaluating prospective trials, as conventional VOI methods can vastly underestimate their value. Trialists and funders can and should assess the VOT metric instead when considering trial designs and costs. If VOI is low, the VOM and cost of a trial can be compared with the benefits and costs of other outreach programs to determine the most cost-effective way to improve uptake.

A Unified Approach for Outliers and Influential Data Detection - The Value of Information in Retrospect

Identifying influential and outlying data is important as it would guide the effective collection of future data and the proper use of existing information. We develop a unified approach for outlier detection and influence analysis. Our proposed method is grounded in the intuitive value of information concepts and has a distinct advantage in interpretability and flexibility when compared to existing methods: it decomposes the data influence into the leverage effect (expected to be influential) and the outlying effect (surprisingly more influential than being expected); and it applies to all decision problems such as estimation, prediction, and hypothesis testing. We study the theoretical properties of three value of information quantities, establish the relationship between the proposed measures and classic measures in the linear regression setting, and provide real data analysis examples of how to apply the new value of information approach in the cases of linear regression, generalized linear mixed model, and hypothesis testing.

Prioritizing Additional Data Collection to Reduce Decision Uncertainty in the HIV/AIDS Response in 6 US Cities: A Value of Information Analysis

OBJECTIVES

The ambitious goals of the US Ending the HIV Epidemic initiative will require a targeted, context-specific public health response. Model-based economic evaluation provides useful guidance for decision making while characterizing decision uncertainty. We aim to quantify the value of eliminating uncertainty about different parameters in selecting combination implementation strategies to reduce the public health burden of HIV/AIDS in 6 US cities and identify future data collection priorities.

METHODS

We used a dynamic compartmental HIV transmission model developed for 6 US cities to evaluate the cost-effectiveness of a range of combination implementation strategies. Using a metamodeling approach with nonparametric and deep learning methods, we calculated the expected value of perfect information, representing the maximum value of further research to eliminate decision uncertainty, and the expected value of partial perfect information for key groups of parameters that would be collected together in practice.

RESULTS

The population expected value of perfect information ranged from $59 683 (Miami) to $54 108 679 (Los Angeles). The rank ordering of expected value of partial perfect information on key groups of parameters were largely consistent across cities and highest for parameters pertaining to HIV risk behaviors, probability of HIV transmission, health service engagement, HIV-related mortality, health utility weights, and healthcare costs. Los Angeles was an exception, where parameters on retention in pre-exposure prophylaxis ranked highest in contributing to decision uncertainty.

CONCLUSIONS

Funding additional data collection on HIV/AIDS may be warranted in Baltimore, Los Angeles, and New York City. Value of information analysis should be embedded into decision-making processes on funding future research and public health intervention.

Practical metrics for establishing the health benefits of research to support research prioritisation

INTRODUCTION

We present practical metrics for estimating the expected health benefits of specific research proposals. These can be used by research funders, researchers and healthcare decision-makers within low-income and middle-income countries to support evidence-based research prioritisation.

METHODS

The methods require three key assessments: (1) the current level of uncertainty around the endpoints the proposed study will measure; (2) how uncertainty impacts on the health benefits and costs of healthcare programmes and (3) the health opportunity costs imposed by programme costs. Research is valuable because it can improve health by informing the choice of which programmes should be implemented. We provide a Microsoft Excel tool to allow readers to generate estimates of the health benefits of research studies based on these three assessments. The tool can be populated using existing studies, existing cost-effectiveness models and expert opinion. Where such evidence is not available, the tool can quantify the value of research under different assumptions. Estimates of the health benefits of research can be considered alongside research costs, and the consequences of delaying implementation until research reports, to determine whether research is worthwhile. We illustrate the method using a case study of research on HIV self-testing programmes in Malawi. This analysis combines data from the literature with outputs from the HIV synthesis model.

RESULTS

For this case study, we found a costing study that could be completed and inform decision making within 1 year offered the highest health benefits (67 000 disability-adjusted life years (DALYs) averted). Research on outcomes improved population health to a lesser extent (12 000 DALYs averted) and only if carried out alongside programme implementation.

CONCLUSION

Our work provides a method for estimating the health benefits of research in a practical and timely fashion. This can be used to support accountable use of research funds.

The Curve of Optimal Sample Size (COSS): A Graphical Representation of the Optimal Sample Size from a Value of Information Analysis

Value of information (VOI) analysis quantifies the opportunity cost associated with decision uncertainty, and thus informs the value of collecting further information to avoid this cost. VOI can inform study design, optimal sample size selection, and research prioritization. Recent methodological advances have reduced the computational burden of conducting VOI analysis and have made it easier to evaluate the expected value of sample information, the expected net benefit of sampling, and the optimal sample size of a study design ([Formula: see text]). The volume of VOI analyses being published is increasing, and there is now a need for VOI studies to conduct sensitivity analyses on VOI-specific parameters. In this practical application, we introduce the curve of optimal sample size (COSS), which is a graphical representation of [Formula: see text] over a range of willingness-to-pay thresholds and VOI parameters (example data and R code are provided). In a single figure, the COSS presents summary data for decision makers to determine the sample size that optimizes research funding given their operating characteristics. The COSS also presents variation in the optimal sample size given variability or uncertainty in VOI parameters. The COSS represents an efficient and additional approach for summarizing results from a VOI analysis.