Dimensions of design space: a decision-theoretic approach to optimal research design

Making Drug Approval Decisions in the Face of Uncertainty: Cumulative Evidence versus Value of Information

BACKGROUND

The COVID-19 pandemic underscored the criticality and complexity of decision making for novel treatment approval and further research. Our study aims to assess potential decision-making methodologies, an evaluation vital for refining future public health crisis responses.

METHODS

We compared 4 decision-making approaches to drug approval and research: the Food and Drug Administration's policy decisions, cumulative meta-analysis, a prospective value-of-information (VOI) approach (using information available at the time of decision), and a reference standard (retrospective VOI analysis using information available in hindsight). Possible decisions were to reject, accept, provide emergency use authorization, or allow access to new therapies only in research settings. We used monoclonal antibodies provided to hospitalized COVID-19 patients as a case study, examining the evidence from September 2020 to December 2021 and focusing on each method's capacity to optimize health outcomes and resource allocation.

RESULTS

Our findings indicate a notable discrepancy between policy decisions and the reference standard retrospective VOI approach with expected losses up to $269 billion USD, suggesting suboptimal resource use during the wait for emergency use authorization. Relying solely on cumulative meta-analysis for decision making results in the largest expected loss, while the policy approach showed a loss up to $16 billion and the prospective VOI approach presented the least loss (up to $2 billion).

CONCLUSION

Our research suggests that incorporating VOI analysis may be particularly useful for research prioritization and treatment implementation decisions during pandemics. While the prospective VOI approach was favored in this case study, further studies should validate the ideal decision-making method across various contexts. This study's findings not only enhance our understanding of decision-making strategies during a health crisis but also provide a potential framework for future pandemic responses.

HIGHLIGHTS

This study reviews discrepancies between a reference standard (retrospective VOI, using hindsight information) and 3 conceivable real-time approaches to research-treatment decisions during a pandemic, suggesting suboptimal use of resources.Of all prospective decision-making approaches considered, VOI closely mirrored the reference standard, yielding the least expected value loss across our study timeline.This study illustrates the possible benefit of VOI results and the need for evidence accumulation accompanied by modeling in health technology assessment for emerging therapies.

Rapid Assessment of the Need for Evidence: Applying the Principles of Value of Information to Research Prioritisation

We propose a short-cut heuristic approach to rapidly estimate value of information (VOI) using information commonly reported in a research funding application to make a case for the need for further evaluative research. We develop a "Rapid VOI" approach, which focuses on uncertainty in the primary outcome of clinical effectiveness and uses this to explore the health consequences of decision uncertainty. We develop a freely accessible online tool, Rapid Assessment of the Need for Evidence (RANE), to allow for the efficient computation of the value of research. As a case study, the method was applied to a proposal for research on shoulder pain rehabilitation. The analysis was included as part of a successful application for research funding to the UK National Institute for Health and Care Research. Our approach enables research funders and applicants to rapidly estimate the value of proposed research. Rapid VOI relies on information that is readily available and reported in research funding applications. Rapid VOI supports research prioritisation and commissioning decisions where there is insufficient time and resources available to develop and validate complex decision-analytic models. The method provides a practical means for implementing VOI in practice, thus providing a starting point for deliberation and contributing to the transparency and accountability of research prioritisation decisions.

Value of Information for Clinical Trial Design: The Importance of Considering All Relevant Comparators

Value of Information (VOI) analyses calculate the economic value that could be generated by obtaining further information to reduce uncertainty in a health economic decision model. VOI has been suggested as a tool for research prioritisation and trial design as it can highlight economically valuable avenues for future research. Recent methodological advances have made it increasingly feasible to use VOI in practice for research; however, there are critical differences between the VOI approach and the standard methods used to design research studies such as clinical trials. We aimed to highlight key differences between the research design approach based on VOI and standard clinical trial design methods, in particular the importance of considering the full decision context. We present two hypothetical examples to demonstrate that VOI methods are only accurate when (1) all feasible comparators are included in the decision model when designing research, and (2) all comparators are retained in the decision model once the data have been collected and a final treatment recommendation is made. Omitting comparators from either the design or analysis phase of research when using VOI methods can lead to incorrect trial designs and/or treatment recommendations. Overall, we conclude that incorrectly specifying the health economic model by ignoring potential comparators can lead to misleading VOI results and potentially waste scarce research resources.

Consolidated Health Economic Evaluation Reporting Standards - Value of Information (CHEERS-VOI): Explanation and Elaboration

OBJECTIVES

Although the ISPOR Value of Information (VOI) Task Force's reports outline VOI concepts and provide good-practice recommendations, there is no guidance for reporting VOI analyses. VOI analyses are usually performed alongside economic evaluations for which the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 Statement provides reporting guidelines. Thus, we developed the CHEERS-VOI checklist to provide reporting guidance and checklist to support the transparent, reproducible, and high-quality reporting of VOI analyses.

METHODS

A comprehensive literature review generated a list of 26 candidate reporting items. These candidate items underwent a Delphi procedure with Delphi participants through 3 survey rounds. Participants rated each item on a 9-point Likert scale to indicate its relevance when reporting the minimal, essential information about VOI methods and provided comments. The Delphi results were reviewed at 2-day consensus meetings and the checklist was finalized using anonymous voting.

RESULTS

We had 30, 25, and 24 Delphi respondents in rounds 1, 2, and 3, respectively. After incorporating revisions recommended by the Delphi participants, all 26 candidate items proceeded to the 2-day consensus meetings. The final CHEERS-VOI checklist includes all CHEERS items, but 7 items require elaboration when reporting VOI. Further, 6 new items were added to report information relevant only to VOI (eg, VOI methods applied).

CONCLUSIONS

The CHEERS-VOI checklist should be used when a VOI analysis is performed alongside economic evaluations. The CHEERS-VOI checklist will help decision makers, analysts and peer reviewers in the assessment and interpretation of VOI analyses and thereby increase transparency and rigor in decision making.

Emerging Therapies for COVID-19: The Value of Information From More Clinical Trials

OBJECTIVES

The COVID-19 pandemic necessitates time-sensitive policy and implementation decisions regarding new therapies in the face of uncertainty. This study aimed to quantify consequences of approving therapies or pursuing further research: immediate approval, use only in research, approval with research (eg, emergency use authorization), or reject.

METHODS

Using a cohort state-transition model for hospitalized patients with COVID-19, we estimated quality-adjusted life-years (QALYs) and costs associated with the following interventions: hydroxychloroquine, remdesivir, casirivimab-imdevimab, dexamethasone, baricitinib-remdesivir, tocilizumab, lopinavir-ritonavir, interferon beta-1a, and usual care. We used the model outcomes to conduct cost-effectiveness and value of information analyses from a US healthcare perspective and a lifetime horizon.

RESULTS

Assuming a $100 000-per-QALY willingness-to-pay threshold, only remdesivir, casirivimab-imdevimab, dexamethasone, baricitinib-remdesivir, and tocilizumab were (cost-) effective (incremental net health benefit 0.252, 0.164, 0.545, 0.668, and 0.524 QALYs and incremental net monetary benefit $25 249, $16 375, $54 526, $66 826, and $52 378). Our value of information analyses suggest that most value can be obtained if these 5 therapies are approved for immediate use rather than requiring additional randomized controlled trials (RCTs) (net value $20.6 billion, $13.4 billion, $7.4 billion, $54.6 billion, and $7.1 billion), hydroxychloroquine (net value $198 million) is only used in further RCTs if seeking to demonstrate decremental cost-effectiveness and otherwise rejected, and interferon beta-1a and lopinavir-ritonavir are rejected (ie, neither approved nor additional RCTs).

CONCLUSIONS

Estimating the real-time value of collecting additional evidence during the pandemic can inform policy makers and clinicians about the optimal moment to implement therapies and whether to perform further research.

Value of Information Analysis in Models to Inform Health Policy

Value of information (VoI) is a decision-theoretic approach to estimating the expected benefits from collecting further information of different kinds, in scientific problems based on combining one or more sources of data. VoI methods can assess the sensitivity of models to different sources of uncertainty and help to set priorities for further data collection. They have been widely applied in healthcare policy making, but the ideas are general to a range of evidence synthesis and decision problems. This article gives a broad overview of VoI methods, explaining the principles behind them, the range of problems that can be tackled with them, and how they can be implemented, and discusses the ongoing challenges in the area.

Simulating Study Data to Support Expected Value of Sample Information Calculations: A Tutorial

The expected value of sample information (EVSI) can be used to prioritize avenues for future research and design studies that support medical decision making and offer value for money spent. EVSI is calculated based on 3 key elements. Two of these, a probabilistic model-based economic evaluation and updating model uncertainty based on simulated data, have been frequently discussed in the literature. By contrast, the third element, simulating data from the proposed studies, has received little attention. This tutorial contributes to bridging this gap by providing a step-by-step guide to simulating study data for EVSI calculations. We discuss a general-purpose algorithm for simulating data and demonstrate its use to simulate 3 different outcome types. We then discuss how to induce correlations in the generated data, how to adjust for common issues in study implementation such as missingness and censoring, and how individual patient data from previous studies can be leveraged to undertake EVSI calculations. For all examples, we provide comprehensive code written in the R language and, where possible, Excel spreadsheets in the supplementary materials. This tutorial facilitates practical EVSI calculations and allows EVSI to be used to prioritize research and design studies.

Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

BACKGROUND

The expected value of sample information (EVSI) calculates the value of collecting additional information through a research study with a given design. However, standard EVSI analyses do not account for the slow and often incomplete implementation of the treatment recommendations that follow research. Thus, standard EVSI analyses do not correctly capture the value of the study. Previous research has developed measures to calculate the research value while adjusting for implementation challenges, but estimating these measures is a challenge.

METHODS

Based on a method that assumes the implementation level is related to the strength of evidence in favor of the treatment, 2 implementation-adjusted EVSI calculation methods are developed. These novel methods circumvent the need for analytical calculations, which were restricted to settings in which normality could be assumed. The first method developed in this article uses computationally demanding nested simulations, based on the definition of the implementation-adjusted EVSI. The second method is based on adapting the moment matching method, a recently developed efficient EVSI computation method, to adjust for imperfect implementation. The implementation-adjusted EVSI is then calculated with the 2 methods across 3 examples.

RESULTS

The maximum difference between the 2 methods is at most 6% in all examples. The efficient computation method is between 6 and 60 times faster than the nested simulation method in this case study and could be used in practice.

CONCLUSIONS

This article permits the calculation of an implementation-adjusted EVSI using realistic assumptions. The efficient estimation method is accurate and can estimate the implementation-adjusted EVSI in practice. By adapting standard EVSI estimation methods, adjustments for imperfect implementation can be made with the same computational cost as a standard EVSI analysis.

HIGHLIGHTS

Standard expected value of sample information (EVSI) analyses do not account for the fact that treatment implementation following research is often slow and incomplete, meaning they incorrectly capture the value of the study.Two methods, based on nested Monte Carlo sampling and the moment matching EVSI calculation method, are developed to adjust EVSI calculations for imperfect implementation when the speed and level of the implementation of a new treatment depends on the strength of evidence in favor of the treatment.The 2 methods we develop provide similar estimates for the implementation-adjusted EVSI.Our methods extend current EVSI calculation algorithms and thus require limited additional computational complexity.

Prioritisation and design of clinical trials

Clinical trials require participation of numerous patients, enormous research resources and substantial public funding. Time-consuming trials lead to delayed implementation of beneficial interventions and to reduced benefit to patients. This manuscript discusses two methods for the allocation of research resources and reviews a framework for prioritisation and design of clinical trials. The traditional error-driven approach of clinical trial design controls for type I and II errors. However, controlling for those statistical errors has limited relevance to policy makers. Therefore, this error-driven approach can be inefficient, waste research resources and lead to research with limited impact on daily practice. The novel value-driven approach assesses the currently available evidence and focuses on designing clinical trials that directly inform policy and treatment decisions. Estimating the net value of collecting further information, prior to undertaking a trial, informs a decision maker whether a clinical or health policy decision can be made with current information or if collection of extra evidence is justified. Additionally, estimating the net value of new information guides study design, data collection choices, and sample size estimation. The value-driven approach ensures the efficient use of research resources, reduces unnecessary burden to trial participants, and accelerates implementation of beneficial healthcare interventions.

Optimal Allocation of Research Funds under a Budget Constraint

Purpose. Health economic evaluations that include the expected value of sample information support implementation decisions as well as decisions about further research. However, just as decision makers must consider portfolios of implementation spending, they must also identify the optimal portfolio of research investments. Methods. Under a fixed research budget, a decision maker determines which studies to fund; additional budget allocated to one study to increase the study sample size implies less budget available to collect information to reduce decision uncertainty in other implementation decisions. We employ a budget-constrained portfolio optimization framework in which the decisions are whether to invest in a study and at what sample size. The objective is to maximize the sum of the studies' population expected net benefit of sampling (ENBS). We show how to determine the optimal research portfolio and study-specific levels of investment. We demonstrate our framework with a stylized example to illustrate solution features and a real-world application using 6 published cost-effectiveness analyses. Results. Among the studies selected for nonzero investment, the optimal sample size occurs at the point at which the marginal population ENBS divided by the marginal cost of additional sampling is the same for all studies. Compared with standard ENBS optimization without a research budget constraint, optimal budget-constrained sample sizes are typically smaller but allow more studies to be funded. Conclusions. The budget constraint for research studies directly implies that the optimal sample size for additional research is not the point at which the ENBS is maximized for individual studies. A portfolio optimization approach can yield higher total ENBS. Ultimately, there is a maximum willingness to pay for incremental information that determines optimal sample sizes.

Calculating the Expected Value of Sample Information in Practice: Considerations from 3 Case Studies

Background. Investing efficiently in future research to improve policy decisions is an important goal. Expected value of sample information (EVSI) can be used to select the specific design and sample size of a proposed study by assessing the benefit of a range of different studies. Estimating EVSI with the standard nested Monte Carlo algorithm has a notoriously high computational burden, especially when using a complex decision model or when optimizing over study sample sizes and designs. Recently, several more efficient EVSI approximation methods have been developed. However, these approximation methods have not been compared, and therefore their comparative performance across different examples has not been explored. Methods. We compared 4 EVSI methods using 3 previously published health economic models. The examples were chosen to represent a range of real-world contexts, including situations with multiple study outcomes, missing data, and data from an observational rather than a randomized study. The computational speed and accuracy of each method were compared. Results. In each example, the approximation methods took minutes or hours to achieve reasonably accurate EVSI estimates, whereas the traditional Monte Carlo method took weeks. Specific methods are particularly suited to problems where we wish to compare multiple proposed sample sizes, when the proposed sample size is large, or when the health economic model is computationally expensive. Conclusions. As all the evaluated methods gave estimates similar to those given by traditional Monte Carlo, we suggest that EVSI can now be efficiently computed with confidence in realistic examples. No systematically superior EVSI computation method exists as the properties of the different methods depend on the underlying health economic model, data generation process, and user expertise.

Value of Information Analytical Methods: Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force

The allocation of healthcare resources among competing priorities requires an assessment of the expected costs and health effects of investing resources in the activities and of the opportunity cost of the expenditure. To date, much effort has been devoted to assessing the expected costs and health effects, but there remains an important need to also reflect the consequences of uncertainty in resource allocation decisions and the value of further research to reduce uncertainty. Decision making with uncertainty may turn out to be suboptimal, resulting in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to better inform resource decisions. This value can be quantified using value of information (VOI) analysis. This report from the ISPOR VOI Task Force describes methods for computing 4 VOI measures: the expected value of perfect information, expected value of partial perfect information (EVPPI), expected value of sample information (EVSI), and expected net benefit of sampling (ENBS). Several methods exist for computing EVPPI and EVSI, and this report provides guidance on selecting the most appropriate method based on the features of the decision problem. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing VOI analyses. The software needed to compute VOI is discussed, and areas for future research are highlighted.

Value of Information Analysis for Research Decisions-An Introduction: Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force

Healthcare resource allocation decisions made under conditions of uncertainty may turn out to be suboptimal. In a resource constrained system in which there is a fixed budget, these suboptimal decisions will result in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to make better resource allocation decisions. This value can be quantified using a value of information (VOI) analysis. This report, from the ISPOR VOI Task Force, introduces VOI analysis, defines key concepts and terminology, and outlines the role of VOI for supporting decision making, including the steps involved in undertaking and interpreting VOI analyses. The report is specifically aimed at those tasked with making decisions about the adoption of healthcare or the funding of healthcare research. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing the results of VOI analyses.

Value of Information Choices that Influence Estimates: A Systematic Review of Prevailing Considerations

Background. Although value of information (VOI) analyses are increasingly advocated and used for research prioritization and reimbursement decisions, the interpretation and usefulness of VOI outcomes depend critically on the underlying choices and assumptions used in the analysis. In this article, we present a structured overview of all items reported in literature to potentially influence VOI outcomes. Use of this overview increases awareness and transparency of choices and assumptions underpinning VOI outcomes. Methods. A systematic literature review was performed to identify aspects of VOI analyses that were found to potentially influence VOI outcomes. Identified aspects were grouped to develop a structured overview. Explanations were defined for all items included in the overview. Results. We retrieved 687 unique papers, of which 71 original papers and 8 reviews were included. In the full text of these 79 papers, 16 aspects were found that may influence VOI outcomes. These aspects related to the underlying evidence (bias, synthesis, heterogeneity, correlation), uncertainty (structural, future pricing), model (relevance, approach, population), choices in VOI calculation (estimation technique, implementation level, population size, perspective), and aspects specifically for assessing the value of future study designs (reversal costs, efficient estimator). These aspects were aggregated into 7 items to provide a structured overview. Conclusion. The developed overview should increase awareness of key choices underlying VOI analysis and facilitate structured reporting of such choices and interpretation of the ensuing VOI outcomes by researchers and policy makers. Use of this overview should improve prioritization and reimbursement decisions.

The Impact of Decision Makers' Constraints on the Outcome of Value of Information Analysis

BACKGROUND

When proven effective, decision making regarding reimbursement of new health technology typically involves ethical, social, legal, and health economic aspects and constraints. Nevertheless, when applying standard value of information (VOI) analysis, the value of collecting additional evidence is typically estimated assuming that only cost-effectiveness outcomes guide such decisions.

OBJECTIVES

To illustrate how decision makers' constraints can be incorporated into VOI analyses and how these may influence VOI outcomes.

METHODS

A simulation study was performed to estimate the cost-effectiveness of a new hypothetical technology compared with usual care. Constraints were defined for the new technology on 1) the maximum acceptable rate of complications and 2) the maximum acceptable additional budget. The expected value of perfect information (EVPI) for the new technology was estimated in various scenarios, both with and without incorporating these constraints.

RESULTS

For a willingness-to-pay threshold of €20,000 per quality-adjusted life-year, the probability that the new technology was cost-effective equaled 57%, with an EVPI of €1868 per patient. Applying the complication rate constraint reduced the EVPI to €1137. Similarly, the EVPI reduced to €770 when applying the budget constraint. Applying both constraints simultaneously further reduced the EVPI to €318.

CONCLUSIONS

When decision makers explicitly apply additional constraints, beyond a willingness-to-pay threshold, to reimbursement decisions, these constraints can and should be incorporated into VOI analysis as well, because they may influence VOI outcomes. This requires continuous interaction between VOI analysts and decision makers and is expected to improve both the relevance and the acceptance of VOI outcomes.

Research Costs Investigated: A Study Into the Budgets of Dutch Publicly Funded Drug-Related Research

BACKGROUND

The costs of performing research are an important input in value of information (VOI) analyses but are difficult to assess.

OBJECTIVE

The aim of this study was to investigate the costs of research, serving two purposes: (1) estimating research costs for use in VOI analyses; and (2) developing a costing tool to support reviewers of grant proposals in assessing whether the proposed budget is realistic.

METHODS

For granted study proposals from the Netherlands Organization for Health Research and Development (ZonMw), type of study, potential cost drivers, proposed budget, and general characteristics were extracted. Regression analysis was conducted in an attempt to generate a 'predicted budget' for certain combinations of cost drivers, for implementation in the costing tool.

RESULTS

Of 133 drug-related research grant proposals, 74 were included for complete data extraction. Because an association between cost drivers and budgets was not confirmed, we could not generate a predicted budget based on regression analysis, but only historic reference budgets given certain study characteristics. The costing tool was designed accordingly, i.e. with given selection criteria the tool returns the range of budgets in comparable studies. This range can be used in VOI analysis to estimate whether the expected net benefit of sampling will be positive to decide upon the net value of future research.

CONCLUSION

The absence of association between study characteristics and budgets may indicate inconsistencies in the budgeting or granting process. Nonetheless, the tool generates useful information on historical budgets, and the option to formally relate VOI to budgets. To our knowledge, this is the first attempt at creating such a tool, which can be complemented with new studies being granted, enlarging the underlying database and keeping estimates up to date.

Efficient Research Design

In designing economic evaluations alongside clinical trials, analysts are frequently faced with alternative methods of collecting the same data, the extremes being top-down (“gross costing”) and bottom-up (“micro-costing”) approaches. A priori, bottom-up approaches may be considered superior to top-down approaches but are also more expensive to collect and analyze. In this article, we use value-of-information analysis to estimate the efficient mix of observations on each method in a proposed clinical trial. By assigning a prior bivariate distribution to the 2 data collection processes, the predicted posterior (i.e., preposterior) mean and variance of the superior process can be calculated from proposed samples using either process. This is then used to calculate the preposterior mean and variance of incremental net benefit and hence the expected net gain of sampling. We apply this method to a previously collected data set to estimate the value of conducting a further trial and identifying the optimal mix of observations on drug costs at 2 levels: by individual item (process A) and by drug class (process B). We find that substituting a number of observations on process A for process B leads to a modest £35,000 increase in expected net gain of sampling. Drivers of the results are the correlation between the 2 processes and their relative cost. This method has potential use following a pilot study to inform efficient data collection approaches for a subsequent full-scale trial. It provides a formal quantitative approach to inform trialists whether it is efficient to collect resource use data on all patients in a trial or on a subset of patients only or to collect limited data on most and detailed data on a subset.

A Practical Application of Value of Information and Prospective Payback of Research to Prioritize Evaluative Research

BACKGROUND AND OBJECTIVES

Efforts to ensure that funded research represents "value for money" have led to increasing calls for the use of analytic methods in research prioritization. A number of analytic approaches have been proposed to assist research funding decisions, the most prominent of which are value of information (VOI) and prospective payback of research (PPoR). Despite the increasing interest in the topic, there are insufficient VOI and PPoR applications on the same case study to contrast their methods and compare their outcomes. We undertook VOI and PPoR analyses to determine the value of conducting 2 proposed research programs. The application served as a vehicle for identifying differences and similarities between the methods, provided insight into the assumptions and practical requirements of undertaking prospective analyses for research prioritization, and highlighted areas for future research.

METHODS

VOI and PPoR were applied to case studies representing proposals for clinical trials in advanced non-small-cell lung cancer and prostate cancer. Decision models were built to synthesize the evidence available prior to the funding decision. VOI (expected value of perfect and sample information) and PPoR (PATHS model) analyses were undertaken using the developed models.

RESULTS AND CONCLUSIONS

VOI and PPoR results agreed in direction, suggesting that the proposed trials would be cost-effective investments. However, results differed in magnitude, largely due to the way each method conceptualizes the possible outcomes of further research and the implementation of research results in practice. Compared with VOI, PPoR is less complex but requires more assumptions. Although the approaches are not free from limitations, they can provide useful input for research funding decisions.

Evaluation of patient registries supporting reimbursement decisions: the case of oxaliplatin for treatment of stage III colon cancer

BACKGROUND

Access with evidence development has been established for expensive intramural drugs in The Netherlands. The procedure involves a 4-year period of conditional reimbursement. During this period, additional evidence has to be gathered usually through a patient registry. Given the costs and time involved in gathering the data, it is important to carefully evaluate the registry.

OBJECTIVES

This study aimed to develop a model for the regular evaluation of patient registries during an access with evidence development process and find the optimal length of the registry period.

METHODS

We used data from a recent registry in The Netherlands on oxaliplatin as a treatment option for stage III colon cancer. We added simulated follow-up data to the empirical data available and applied value of information analysis to balance the gains of extending the period and amount of data gathering against the costs of registering patients.

RESULTS

We show that given the assumptions on cohort size, follow-up time, and purpose of the registry, the current (partly simulated) registry was not very efficient. Notably, the observation period could have been stopped to make a definite reimbursement decision after a maximum of 2 years rather than the fixed 4-year period.

CONCLUSIONS

Patient registries may be an efficient way to gather data on new medical treatments, but they need to be carefully designed and evaluated, in particular regarding their follow-up time. For each purpose, data gathering can be tailored to make sure decisions are taken at the moment that sufficient data are available.

Design of implementation studies for quality improvement programs: an effectiveness-cost-effectiveness framework

Translational research applies basic science discoveries in clinical and community settings. Implementation research is often limited by tremendous variability among settings; therefore, generalization of findings may be limited. Adoption of a novel procedure in a community practice is usually a local decision guided by setting-specific knowledge. The conventional statistical framework that aims to produce generalizable knowledge is inappropriate for local quality improvement investigations. We propose an analytic framework based on cost-effectiveness of the implementation study design, taking into account prior knowledge from local experts. When prior knowledge does not indicate a clear preference between the new and standard procedures, local investigation should guide the choice. The proposed approach requires substantially smaller sample sizes than the conventional approach. Sample size formulae and general guidance are provided.

Optimal decision rules for HTA under uncertainty: a wider, dynamic perspective

We present a two-period framework, which combines real option and decision-theoretic approaches to health technology assessment under uncertainty. By viewing adoption, treatment and research decisions as a single economic project, we illustrate how their key dimensions affect optimal rules. We consider the results in relation to the existing literature and argue that developments in this direction could contribute substantially to efficiency gains in resource allocation.

Value of information analysis in oncology: the value of evidence and evidence of value

PURPOSE

Value of information (VOI) analysis is a novel systematic approach for assessing whether there is sufficient evidence to support regulatory approval of new technologies, estimating the value of additional research, informing trial design, and setting research priorities. This article reviews the use of VOI methods in oncology and identifies the potential applications of VOI in this field.

METHODS

A systematic literature search was undertaken to identify studies explicitly reporting VOI analyses for interventions directed at cancer management. Articles published from 2000 onward addressing prevention, screening, diagnosis, treatment, or symptom management in oncology were selected.

RESULTS

A total of 35 articles were included in the review; most were published after 2006. The main cancers addressed were breast (n = 10; 29%), prostate (n = 5; 14%), lung (n = 5; 14%), and colorectal (n = 3; 9%). The VOI analyses were of an applied nature in 31 studies (89%). In the applied studies, VOI was used to characterize decision uncertainty in all studies and to inform future research focus in 16 (52%). Additionally, one article (3%) addressed the value of optimal trial design, and one article (3%) reported the use of VOI methods to prioritize research.

CONCLUSION

The application of VOI analysis in oncology is growing but remains limited. Benefits in oncology research and practice will potentially be optimized with an increase in the application of VOI methods to inform decision making, optimal trial design, and research prioritization in this field.

A systematic and critical review of the evolving methods and applications of value of information in academia and practice

OBJECTIVE

This article provides a systematic and critical review of the evolving methods and applications of value of information (VOI) in academia and practice and discusses where future research needs to be directed.

METHODS

Published VOI studies were identified by conducting a computerized search on Scopus and ISI Web of Science from 1980 until December 2011 using pre-specified search terms. Only full-text papers that outlined and discussed VOI methods for medical decision making, and studies that applied VOI and explicitly discussed the results with a view to informing healthcare decision makers, were included. The included papers were divided into methodological and applied papers, based on the aim of the study.

RESULTS

A total of 118 papers were included of which 50 % (n = 59) are methodological. A rapidly accumulating literature base on VOI from 1999 onwards for methodological papers and from 2005 onwards for applied papers is observed. Expected value of sample information (EVSI) is the preferred method of VOI to inform decision making regarding specific future studies, but real-life applications of EVSI remain scarce. Methodological challenges to VOI are numerous and include the high computational demands, dealing with non-linear models and interdependency between parameters, estimations of effective time horizons and patient populations, and structural uncertainties.

CONCLUSION

VOI analysis receives increasing attention in both the methodological and the applied literature bases, but challenges to applying VOI in real-life decision making remain. For many technical and methodological challenges to VOI analytic solutions have been proposed in the literature, including leaner methods for VOI. Further research should also focus on the needs of decision makers regarding VOI.

Coverage with evidence development, only in research, risk sharing, or patient access scheme? A framework for coverage decisions

BACKGROUND

Until recently, purchasers' options regarding whether to pay for the use of medical technologies have been binary in nature: a treatment is either covered or not. Policies, however, have emerged that expand the decision options, for example, linking coverage to evidence development, an option increasingly used for treatments with limited/uncertain evidence. There has been little effort to reconcile the features of technologies with the available decision options.

METHODS

We described a framework within which different decision options can be evaluated. We distinguished two sources of value in terms of health: the value of the technology per se and the value of reducing decision uncertainty. The costs of reversing decisions were also considered.

FINDINGS

Purchasers should weigh the expected benefits of coverage against the possibility that the decision may need to be reversed and the chance that adoption will hinder evidence generation. Based on the purchaser's range of authority over access, research, and price and on the characteristics of the technology with regard to reversibility and evidence, different decisions may be appropriate. The framework clarified the assessments needed to establish the appropriateness of different decisions. A taxonomy of coverage decisions was suggested.

CONCLUSIONS

A range of decision options may facilitate paying for the use of promising medical technologies despite their uncertain evidence. It is important that the option be chosen on the basis of not only the expected value of a technology but also the value of further research, the anticipated effect of coverage on further research, and the costs associated with reversing the decision.

A CTSA agenda to advance methods for comparative effectiveness research

Clinical research needs to be more useful to patients, clinicians, and other decision makers. To meet this need, more research should focus on patient-centered outcomes, compare viable alternatives, and be responsive to individual patients' preferences, needs, pathobiology, settings, and values. These features, which make comparative effectiveness research (CER) fundamentally patient-centered, challenge researchers to adopt or develop methods that improve the timeliness, relevance, and practical application of clinical studies. In this paper, we describe 10 priority areas that address 3 critical needs for research on patient-centered outcomes (PCOR): (1) developing and testing trustworthy methods to identify and prioritize important questions for research; (2) improving the design, conduct, and analysis of clinical research studies; and (3) linking the process and outcomes of actual practice to priorities for research on patient-centered outcomes. We argue that the National Institutes of Health, through its clinical and translational research program, should accelerate the development and refinement of methods for CER by linking a program of methods research to the broader portfolio of large, prospective clinical and health system studies it supports. Insights generated by this work should be of enormous value to PCORI and to the broad range of organizations that will be funding and implementing CER.

Addressing Adoption and Research Design Decisions Simultaneously

Methods to estimate the cost-effectiveness of technologies are well developed with increasing experience of their application to inform adoption decisions in a timely way. However, the experience of using similarly explicit methods to inform the associated research decisions is less well developed despite appropriate methods being available with an increasing number of applications in health. The authors demonstrate that evaluation of both adoption and research decisions is feasible within typical time and resource constraints relevant to policy decisions, even in situations in which data are sparse and formal elicitation is required. In addition to demonstrating the application of expected value of sample information (EVSI) in these circumstances, the authors examine and carefully distinguish the impact that the research decision is expected to have on patients while enrolled in the trial, those not enrolled, and once the trial reports. In doing so, the authors are able to account for the range of opportunity cost associated with research and evaluate a number of research designs including length of follow-up and sample size. The authors also explore the implications for research design of conducting research while the technology is approved for widespread use and whether approval should be withheld until research reports. In doing so, the authors highlight the impact of irrecoverable opportunity costs when the initial costs of a technology are compensated only by later gains in health outcome.