Value-of-information analysis within a stakeholder-driven research prioritization process in a US setting: an application in cancer genomics

Constructing Relative Effect Priors for Research Prioritization and Trial Design: A Meta-epidemiological Analysis

BACKGROUND

Bayesian methods have potential for efficient design of randomized clinical trials (RCTs) by incorporating existing evidence. Furthermore, value of information (VOI) methods estimate the value of reducing decision uncertainty, aiding transparent research prioritization. These methods require a prior distribution describing current uncertainty in key parameters, such as relative treatment effect (RTE). However, at the time of designing and commissioning research, there may be no data to base the prior on. The aim of this article is to present methods to construct priors for RTEs based on a collection of previous RCTs.

METHODS

We developed 2 Bayesian hierarchical models that captured variability in RTE between studies within disease area accounting for study characteristics. We illustrate the methods using a data set of 743 published RCTs across 9 disease areas to obtain predictive distributions for RTEs for a range of disease areas. We illustrate how the priors from such an analysis can be used in a VOI analysis for an RCT in bladder cancer and compare the results with those using an uninformative prior.

RESULTS

For most disease areas, the predicted RTE favored new interventions over comparators. The predicted effects and uncertainty differed across the 9 disease areas. VOI analysis showed that the expected value of research is much lower with our empirically derived prior compared with an uninformative prior.

CONCLUSIONS

This study demonstrates a novel approach to generating informative priors that can be used to aid research prioritization and trial design. The methods can also be used to combine RCT evidence with expert opinion. Further work is needed to create a rich database of RCT evidence that can be used to form off-the-shelf priors.

HIGHLIGHTS

Bayesian methods have potential to aid the efficient design of randomized clinical trials (RCTs) by incorporating existing evidence. Value-of-information (VOI) methods can be used to aid research prioritization by calculating the value of current decision uncertainty.These methods require a distribution describing current uncertainty in key parameters, that is, "prior distributions."This article demonstrates a methodology to estimate prior distributions for relative treatment effects (odds and hazard ratios) estimated from a collection of previous RCTs.These results may be combined with expert elicitation to facilitate 1) value-of-information methods to prioritize research or 2) Bayesian methods for research design.

Approaches to prioritising research for clinical trial networks: a scoping review

BACKGROUND

Prioritisation of clinical trials ensures that the research conducted meets the needs of stakeholders, makes the best use of resources and avoids duplication. The aim of this review was to identify and critically appraise approaches to research prioritisation applicable to clinical trials, to inform best practice guidelines for clinical trial networks and funders.

METHODS

A scoping review of English-language published literature and research organisation websites (January 2000 to January 2020) was undertaken to identify primary studies, approaches and criteria for research prioritisation. Data were extracted and tabulated, and a narrative synthesis was employed.

RESULTS

Seventy-eight primary studies and 18 websites were included. The majority of research prioritisation occurred in oncology and neurology disciplines. The main reasons for prioritisation were to address a knowledge gap (51 of 78 studies [65%]) and to define patient-important topics (28 studies, [35%]). In addition, research organisations prioritised in order to support their institution's mission, invest strategically, and identify best return on investment. Fifty-seven of 78 (73%) studies used interpretative prioritisation approaches (including Delphi surveys, James Lind Alliance and consensus workshops); six studies used quantitative approaches (8%) such as prospective payback or value of information (VOI) analyses; and 14 studies used blended approaches (18%) such as nominal group technique and Child Health Nutritional Research Initiative. Main criteria for prioritisation included relevance, appropriateness, significance, feasibility and cost-effectiveness.

CONCLUSION

Current research prioritisation approaches for groups conducting and funding clinical trials are largely interpretative. There is an opportunity to improve the transparency of prioritisation through the inclusion of quantitative approaches.

Implementation Barriers to Value of Information Analysis in Health Technology Decision Making: Results From a Process Evaluation

OBJECTIVES

Value of information (VOI) analysis can support health technology assessment decision making, but it is a long way from being standard use. The objective of this study was to understand barriers to the implementation of VOI analysis and propose actions to overcome these.

METHODS

We performed a process evaluation of VOI analysis use within decision making on tomosynthesis versus digital mammography for use in the Dutch breast cancer population screening. Based on steering committee meeting attendance and regular meetings with analysts, we developed a list of barriers to VOI use, which were analyzed using an established diffusion model. We proposed actions to address these barriers. Barriers and actions were discussed and validated in a workshop with stakeholders representing patients, clinicians, regulators, policy advisors, researchers, and the industry.

RESULTS

Consensus was reached on groups of barriers, which included characteristics of VOI analysis itself, stakeholder's attitudes, analysts' and policy makers' skills and knowledge, system readiness, and implementation in the organization. Observed barriers did not only pertain to VOI analysis itself but also to formulating the objective of the assessment, economic modeling, and broader aspects of uncertainty assessment. Actions to overcome these barriers related to organizational changes, knowledge transfer, cultural change, and tools.

CONCLUSIONS

This in-depth analysis of barriers to implementation of VOI analysis and resulting actions and tools may be useful to health technology assessment organizations that wish to implement VOI analysis in technology assessment and research prioritization. Further research should focus on application and evaluation of the proposed actions in real-world assessment processes.

Health Economics Tools and Precision Medicine: Opportunities and Challenges

Precision medicine - individualizing care for patients and addressing variations in treatment response - is likely to be important in improving the nation's health in a cost-effective manner. Despite this promise, widespread use of precision medicine, specifically genomic markers, in clinical care has been limited in practice to date. Lack of evidence, clear evidence thresholds, and reimbursement have been cited as major barriers. Health economics frameworks and tools can elucidate the effects of legal, regulatory, and reimbursement policies on the use of precision medicine while guiding research investments to enhance the appropriate use of precision medicine. Despite the capacity of economics to enhance the clinical and human impact of precision medicine, application of health economics to precision medicine has been limited - in part because precision medicine is a relatively new field - but also because precision medicine is complex, both in terms of its applications and implications throughout medicine and the healthcare system. The goals of this review are several-fold: (1) provide an overview of precision medicine and key policy challenges for the field; (2) explain the potential utility of economics methods in addressing these challenges; (3) describe recent research activities; and (4) summarize opportunities for cross-disciplinary research.

Value of Information Analysis for Research Decisions-An Introduction: Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force

Healthcare resource allocation decisions made under conditions of uncertainty may turn out to be suboptimal. In a resource constrained system in which there is a fixed budget, these suboptimal decisions will result in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to make better resource allocation decisions. This value can be quantified using a value of information (VOI) analysis. This report, from the ISPOR VOI Task Force, introduces VOI analysis, defines key concepts and terminology, and outlines the role of VOI for supporting decision making, including the steps involved in undertaking and interpreting VOI analyses. The report is specifically aimed at those tasked with making decisions about the adoption of healthcare or the funding of healthcare research. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing the results of VOI analyses.

Optimal Information Collection Policies in a Markov Decision Process Framework

BACKGROUND

The cost-effectiveness and value of additional information about a health technology or program may change over time because of trends affecting patient cohorts and/or the intervention. Delaying information collection even for parameters that do not change over time may be optimal.

METHODS

We present a stochastic dynamic programming approach to simultaneously identify the optimal intervention and information collection policies. We use our framework to evaluate birth cohort hepatitis C virus (HCV) screening. We focus on how the presence of a time-varying parameter (HCV prevalence) affects the optimal information collection policy for a parameter assumed constant across birth cohorts: liver fibrosis stage distribution for screen-detected diagnosis at age 50.

RESULTS

We prove that it may be optimal to delay information collection until a time when the information more immediately affects decision making. For the example of HCV screening, given initial beliefs, the optimal policy (at 2010) was to continue screening and collect information about the distribution of liver fibrosis at screen-detected diagnosis in 12 years, increasing the expected incremental net monetary benefit (INMB) by $169.5 million compared to current guidelines.

CONCLUSIONS

The option to delay information collection until the information is sufficiently likely to influence decisions can increase efficiency. A dynamic programming framework enables an assessment of the marginal value of information and determines the optimal policy, including when and how much information to collect.

Integrating value of research into NCI Clinical Trials Cooperative Group research review and prioritization: A pilot study

Background

The Institute of Medicine has called for approaches to help maximize the return on investments (ROI) in cancer clinical trials. Value of Research (VOR) is a health economics technique that estimates ROI and can inform research prioritization. Our objective was to evaluate the impact of using VOR analyses on the clinical trial proposal review process within the SWOG cancer clinical trials consortium.

Methods

We used a previously developed minimal modeling approach to calculate VOR estimates for 9 phase II/III SWOG proposals between February 2015 and December 2016. Estimates were presented to executive committee (EC) members (N = 12) who determine which studies are sent to the National Cancer Institute for funding consideration. EC members scored proposals from 1 (best) to 5 based on scientific merit and potential impact before and after receiving VOR estimates. EC members were surveyed to assess research priorities, proposal evaluation process satisfaction, and the VOR process.

Results

Value of Research estimates ranged from −$2.1B to $16.46B per proposal. Following review of VOR results, the EC changed their score for eight of nine proposals. Proposal rankings were different in pre‐ vs postscores (P value: 0.03). Respondents had mixed views of the ultimate utility of VOR for their decisions with most supporting (42%) or neutral (41%) to the idea of adding VOR to the evaluation process.

Conclusions

The findings from this pilot study indicate use of VOR analyses may be a useful adjunct to inform proposal reviews within NCI Cooperative Clinical Trials groups.

The premarket assessment of the cost-effectiveness of a predictive technology "Straticyte™" for the early detection of oral cancer: a decision analytic model

INTRODUCTION

Approximately half of oral cancers are detected in advanced stages. The current gold standard is histopathological assessment of biopsied tissue, which is subjective and dependent on expertise. Straticyte™, a novel prognostic tool at the pre-market stage, that more accurately identifies patients at high risk for oral cancer than histopathology alone. This study conducts an early cost-effectiveness analysis (CEA) of Straticyte™ and histopathology versus histopathology alone for oral cancer diagnosis in adult patients.

METHODS

A decision-analytic model was constructed after narrowing the scope of Straticyte™, and defining application paths. Data was gathered using the belief elicitation method, and systematic review and meta-analysis. The early CEA was conducted from private-payer and patient perspectives, capturing both direct and indirect costs over a five-year time horizon. One-way and probabilistic sensitivity analyses were conducted to investigate uncertainty.

RESULTS

Compared to histopathology alone, histopathology with Straticyte™ was the dominant strategy, resulting in fewer cancer cases (31 versus 36 per 100 patients) and lower total costs per cancer case avoided (3,360 versus 3,553). This remained robust when Straticyte™ was applied to moderate and mild cases, but became slightly more expensive but still more effective than histopathology alone when Straticyte™ was applied to only mild cases. The probabilistic and one-way sensitivity analyses demonstrated that incorporating Straticyte™ to the current algorithm would be cost-effective over a wide range of parameters and willingness-to-pay values.

CONCLUSION

This study demonstrates high probability that Straticyte™ and histopathology will be cost-effective, which encourages continued investment in the product. The analysis is informed by limited clinical data on Straticyte™, however as more data becomes available, more precise estimates will be generated.

Are Evidence Standards Different for Genomic- vs. Clinical-Based Precision Medicine? A Quantitative Analysis of Individualized Warfarin Therapy

Evidence requirements for implementation of precision medicine (PM), whether informed by genomic or clinical data, are not well defined. Evidence requirements are driven by uncertainty and its attendant consequences; these aspects can be quantified by a novel technique in health economics: value of information analysis (VOI). We utilized VOI analysis to compare the evidence levels over time for warfarin dosing based on pharmacogenomic vs. amiodarone-warfarin drug-drug interaction information. The primary outcome was the expected value of perfect information (EVPI), which is an estimate of the upper limit of the societal value of conducting future research. Over the past decade, the EVPI for the pharmacogenomic strategy decreased from $1,550 to $140 vs. $1,220 to $280 per patient for the drug-interaction strategy. Evidence levels thus appear to be higher for pharmacogenomic-guided vs. drug-interaction-guided warfarin dosing. Clinical guidelines and reimbursement policies for warfarin PM could be informed by these findings.

Burden of illness and research investments in translational sciences for pharmaceuticals in metastatic cancers

AIM

To explore whether investments in translational sciences for six metastatic cancers follow idiosyncratic returns to those investments rather than levels of burden of illness (BI).

METHODS

Associate the number of translational clinical trials in the USA involving oncolytic drugs approved during 2008-2013 and the level (in 2008) and changes (2002-2008 and 2008-2014) in cancer-specific years of life lost.

RESULTS

Investments in trials were positively associated only with contemporary changes in BI (2008-2014). The relationship was stronger for government-sponsored comparative-effectiveness trials than for industry.

CONCLUSION

Translational research investments follow anticipated changes to BI levels. Systematic quantification of these expected returns from specific investments can help guide investment decisions in translational health sciences and generate productive dialogue across stakeholders.

Value of Information Analysis Informing Adoption and Research Decisions in a Portfolio of Health Care Interventions

Background: Value of information (VOI) analysis quantifies the value of additional research in reducing decision uncertainty. It addresses adoption and research decisions simultaneously by comparing the expected benefits and costs of research studies. Nevertheless, the application of this approach in practice remains limited. Objectives: To apply VOI analysis in health care interventions to guide adoption decisions, optimize trial design, and prioritize research. Methods: The analysis was from the perspective of Queensland Health, Australia. It included four interventions: clinically indicated catheter replacement, tissue adhesive for securing catheters, negative pressure wound therapy (NPWT) in caesarean sections, and nutritional support for preventing pressure ulcers. For each intervention, cost-effectiveness analysis was performed, decision uncertainty characterized, and VOI calculated using Monte Carlo simulations. The benefits and costs of additional research were considered together with the costs and consequences of acting now versus waiting for more information. All values are reported in 2014 Australian dollars (AU$). Results: All interventions were cost-effective, but with various levels of decision uncertainty. The current evidence is sufficient to support the adoption of clinically indicated catheter replacement. For the tissue adhesive, an additional study before adoption is worthwhile with a four-arm trial of 220 patients per arm. Additional research on NPWT before adoption is worthwhile with a two-arm trial of 200 patients per arm. Nutritional support should be adopted with a two-arm trial of 1200 patients per arm. Based on the expected net monetary benefits, the studies were ranked as follows: 1) NPWT (AU$1.2 million), 2) tissue adhesive (AU$0.3 milliion), and 3) nutritional support (AU$0.1 million). Conclusions: VOI analysis is a useful and practical approach to inform adoption and research decisions. Efforts should be focused on facilitating its integration into decision making frameworks.

Development and Evaluation of an Approach to Using Value of Information Analyses for Real-Time Prioritization Decisions Within SWOG, a Large Cancer Clinical Trials Cooperative Group

OBJECTIVE

Value of information (VOI) analyses can align research with areas with the greatest potential impact on patient outcome, but questions remain concerning the feasibility and acceptability of these approaches to inform prioritization decisions. Our objective was to develop a process for calculating VOI in "real time" to inform trial funding decisions within SWOG, a large cancer clinical trials group.

METHODS

We developed an efficient and scalable VOI modeling approach using a selected sample of 9 randomized phase II/III trial proposals from the Breast, Gastrointestinal, and Genitourinary Disease Committees reviewed by SWOG's leadership between 2008 and 2013. There was bidirectional communication between SWOG investigators and the research team throughout the modeling development. Partial expected value of sample information for the treatment effect evaluated by the proposed trial's primary endpoint was calculated using Monte Carlo simulation.

RESULTS

We derived prior uncertainty in the treatment effect estimate from the sample size calculations. Our process was feasible for 8 of 9 trial proposals and efficient: the time required of 1 researcher was <1 week per proposal. We accommodated stakeholder input primarily by deconstructing VOI metrics into expected health benefits and incremental healthcare costs and assuming treatment decisions within our simulations were based on health benefits. Following customization, feedback from over 200 SWOG members was positive regarding the overall VOI framework, specific retrospective results, and potential for VOI analyses to inform future trial proposal evaluations.

CONCLUSIONS

We developed an efficient and customized process to calculate the expected VOI of cancer clinical trials that is feasible for use in decision making and acceptable to investigators. Prospective use and evaluation of this approach is currently underway within SWOG.

Adjusting Estimates of the Expected Value of Information for Implementation

Background: Value of information (VoI) calculations give the expected benefits of decision making under perfect information (EVPI) or sample information (EVSI), typically on the premise that any treatment recommendations made in light of this information will be implemented instantly and fully. This assumption is unlikely to hold in health care; evidence shows that obtaining further information typically leads to “improved” rather than “perfect” implementation. Objectives: To present a method of calculating the expected value of further research that accounts for the reality of improved implementation. Methods: This work extends an existing conceptual framework by introducing additional states of the world regarding information (sample information, in addition to current and perfect information) and implementation (improved implementation, in addition to current and optimal implementation). The extension allows calculating the “implementation-adjusted” EVSI (IA-EVSI), a measure that accounts for different degrees of implementation. Calculations of implementation-adjusted estimates are illustrated under different scenarios through a stylized case study in non–small cell lung cancer. Results: In the particular case study, the population values for EVSI and IA-EVSI were £25 million and £8 million, respectively; thus, a decision assuming perfect implementation would have overestimated the expected value of research by about £17 million. IA-EVSI was driven by the assumed time horizon and, importantly, the specified rate of change in implementation: the higher the rate, the greater the IA-EVSI and the lower the difference between IA-EVSI and EVSI. Conclusions: Traditionally calculated measures of population VoI rely on unrealistic assumptions about implementation. This article provides a simple framework that accounts for improved, rather than perfect, implementation and offers more realistic estimates of the expected value of research.

Value of Information: A Tool to Improve Research Prioritization and Reduce Waste

In a Guest Editorial, Cosetta Minelli and Gianluca Baio explain how VOI analysis can prioritize research projects by identifying uncertainty in existing knowledge and then estimating expected benefits from reducing that uncertainty.

Squaring the circle: a priority-setting method for evidence-based service development, reconciling research with multiple stakeholder views

BACKGROUND

This study demonstrates a technique to aid the implementation of research findings through an example of improving services and self-management in longer-term depression. In common with other long-term conditions, policy in this field requires innovation to be undertaken in the context of a whole system of care, be cost-effective, evidence-based and to comply with national clinical guidelines. At the same time, successful service development must be acceptable to clinicians and service users and choices must be made within limited resources. This paper describes a novel way of resolving these competing requirements by reconciling different sources and types of evidence and systematically engaging multiple stakeholder views.

METHODS

The study combined results from mathematical modelling of the care pathway, research evidence on effective interventions and findings from qualitative research with service users in a series of workshops to define, refine and select candidate service improvements. A final consensus-generating workshop used structured discussion and anonymised electronic voting. This was followed by an email survey to all stakeholders, to achieve a pre-defined criterion of consensus for six suggestions for implementation.

RESULTS

An initial list of over 20 ideas was grouped into four main areas. At the final workshop, each idea was presented in person, visually and in writing to 40 people, who assigned themselves to one or more of five stakeholder groups: i) service users and carers, ii) clinicians, iii) managers, iv) commissioners and v) researchers. Many belonged to more than one group. After two rounds of voting, consensus was reached on seven ideas and one runner up. The survey then confirmed the top six ideas to be tested in practice.

CONCLUSIONS

The method recruited and retained people with diverse experience and views within a health community and took account of a full range of evidence. It enabled a diverse group of stakeholders to travel together in a direction that converged with the messages coming out of the research and successfully yielded priorities for service improvement that met competing requirements.

A Practical Application of Value of Information and Prospective Payback of Research to Prioritize Evaluative Research

BACKGROUND AND OBJECTIVES

Efforts to ensure that funded research represents "value for money" have led to increasing calls for the use of analytic methods in research prioritization. A number of analytic approaches have been proposed to assist research funding decisions, the most prominent of which are value of information (VOI) and prospective payback of research (PPoR). Despite the increasing interest in the topic, there are insufficient VOI and PPoR applications on the same case study to contrast their methods and compare their outcomes. We undertook VOI and PPoR analyses to determine the value of conducting 2 proposed research programs. The application served as a vehicle for identifying differences and similarities between the methods, provided insight into the assumptions and practical requirements of undertaking prospective analyses for research prioritization, and highlighted areas for future research.

METHODS

VOI and PPoR were applied to case studies representing proposals for clinical trials in advanced non-small-cell lung cancer and prostate cancer. Decision models were built to synthesize the evidence available prior to the funding decision. VOI (expected value of perfect and sample information) and PPoR (PATHS model) analyses were undertaken using the developed models.

RESULTS AND CONCLUSIONS

VOI and PPoR results agreed in direction, suggesting that the proposed trials would be cost-effective investments. However, results differed in magnitude, largely due to the way each method conceptualizes the possible outcomes of further research and the implementation of research results in practice. Compared with VOI, PPoR is less complex but requires more assumptions. Although the approaches are not free from limitations, they can provide useful input for research funding decisions.

Value of Information Analysis Applied to the Economic Evaluation of Interventions Aimed at Reducing Juvenile Delinquency: An Illustration

Objectives

To investigate whether a value of information analysis, commonly applied in health care evaluations, is feasible and meaningful in the field of crime prevention.

Methods

Interventions aimed at reducing juvenile delinquency are increasingly being evaluated according to their cost-effectiveness. Results of cost-effectiveness models are subject to uncertainty in their cost and effect estimates. Further research can reduce that parameter uncertainty. The value of such further research can be estimated using a value of information analysis, as illustrated in the current study. We built upon an earlier published cost-effectiveness model that demonstrated the comparison of two interventions aimed at reducing juvenile delinquency. Outcomes were presented as costs per criminal activity free year.

Results

At a societal willingness-to-pay of €71,700 per criminal activity free year, further research to eliminate parameter uncertainty was valued at €176 million. Therefore, in this illustrative analysis, the value of information analysis determined that society should be willing to spend a maximum of €176 million in reducing decision uncertainty in the cost-effectiveness of the two interventions. Moreover, the results suggest that reducing uncertainty in some specific model parameters might be more valuable than in others.

Conclusions

Using a value of information framework to assess the value of conducting further research in the field of crime prevention proved to be feasible. The results were meaningful and can be interpreted according to health care evaluation studies. This analysis can be helpful in justifying additional research funds to further inform the reimbursement decision in regard to interventions for juvenile delinquents.

Quantitative benefit-harm assessment for setting research priorities: the example of roflumilast for patients with COPD

BACKGROUND

When faced with uncertainties about the effects of medical interventions regulatory agencies, guideline developers, clinicians, and researchers commonly ask for more research, and in particular for more randomized trials. The conduct of additional randomized trials is, however, sometimes not the most efficient way to reduce uncertainty. Instead, approaches such as value of information analysis or other approaches should be used to prioritize research that will most likely reduce uncertainty and inform decisions.

DISCUSSION

In situations where additional research for specific interventions needs to be prioritized, we propose the use of quantitative benefit-harm assessments that illustrate how the benefit-harm balance may change as a consequence of additional research. The example of roflumilast for patients with chronic obstructive pulmonary disease shows that additional research on patient preferences (e.g., how important are exacerbations relative to psychiatric harms?) or outcome risks (e.g., what is the incidence of psychiatric outcomes in patients with chronic obstructive pulmonary disease without treatment?) is sometimes more valuable than additional randomized trials. We propose that quantitative benefit-harm assessments have the potential to explore the impact of additional research and to identify research priorities Our approach may be seen as another type of value of information analysis and as a useful approach to stimulate specific new research that has the potential to change current estimates of the benefit-harm balance and decision making.

Interpretation of the Expected Value of Perfect Information and Research Recommendations

Background. Expected value of perfect information (EVPI) calculations are increasingly performed to guide and underpin research recommendations. An EVPI value that exceeds the estimated cost of research forms a necessary (although not sufficient) condition for further research to be considered worthwhile. However, it is unclear what factors affect researchers’ recommendations and whether there is a notional threshold of positive returns below which research is not recommended. The objectives of this study were to explore whether EVPI and other factors have a bearing on research recommendations and to assess whether there exists a threshold EVPI below which research is typically not recommended. Methods. A systematic literature review was undertaken to identify applied EVPI calculations in the health care field. Study characteristics were extracted, including funder, location, disease group, publication year, primary language, and outcome measure. Population EVPI values and willingness-to-pay thresholds were also extracted alongside verbatim text excerpts describing the authors’ research recommendations. Recommendations were classified according to whether further research was recommended (a positive recommendation) or not (negative). Factors affecting the likelihood of a positive recommendation were examined statistically using logistic regression and visually by plotting the results in graphs. Results and Conclusions. Eighty-six articles were included, of which 13 suggested no further research, 66 recommended further research, and 7 gave no recommendation. EVPI appears to be a key driver of researchers’ recommendations for further research. Disease area, funder, study location, publication year, and outcome may have a bearing on recommendations, although none of these factors reached statistical significance. A threshold EVPI value below which research is typically not recommended was found at around £1.48 million.

Cost-effectiveness analysis alongside clinical trials II-An ISPOR Good Research Practices Task Force report

Clinical trials evaluating medicines, medical devices, and procedures now commonly assess the economic value of these interventions. The growing number of prospective clinical/economic trials reflects both widespread interest in economic information for new technologies and the regulatory and reimbursement requirements of many countries that now consider evidence of economic value along with clinical efficacy. As decision makers increasingly demand evidence of economic value for health care interventions, conducting high-quality economic analyses alongside clinical studies is desirable because they broaden the scope of information available on a particular intervention, and can efficiently provide timely information with high internal and, when designed and analyzed properly, reasonable external validity. In 2005, ISPOR published the Good Research Practices for Cost-Effectiveness Analysis Alongside Clinical Trials: The ISPOR RCT-CEA Task Force report. ISPOR initiated an update of the report in 2014 to include the methodological developments over the last 9 years. This report provides updated recommendations reflecting advances in several areas related to trial design, selecting data elements, database design and management, analysis, and reporting of results. Task force members note that trials should be designed to evaluate effectiveness (rather than efficacy) when possible, should include clinical outcome measures, and should obtain health resource use and health state utilities directly from study subjects. Collection of economic data should be fully integrated into the study. An incremental analysis should be conducted with an intention-to-treat approach, complemented by relevant subgroup analyses. Uncertainty should be characterized. Articles should adhere to established standards for reporting results of cost-effectiveness analyses. Economic studies alongside trials are complementary to other evaluations (e.g., modeling studies) as information for decision makers who consider evidence of economic value along with clinical efficacy when making resource allocation decisions.

Multilevel modeling and value of information in clinical trial decision support

Background

Clinical trials are the main method for evaluating safety and efficacy of medical interventions and have produced many advances in improving human health. The Women’s Health Initiative overturned a half-century of harmful practice in hormone therapy, the National Lung Screening Trial identified the first successful lung cancer screening tool and the Prostate, Lung, Colorectal and Ovarian Cancer Screening Trial overturned decades-long assumptions. While some trials identify unforeseen safety issues or harms, many fail to demonstrate efficacy. Large trials require substantial resources; to ensure reliable outcomes, we must seek ways to improve the predictive information used as the basis of trials.

Results

Here we demonstrate a modeling framework for linking knowledge of underlying biological mechanism to evaluate the expectation of trial outcomes. Key features include the ability to propagate uncertainty in biological mechanism to uncertainty in trial outcome and mechanisms for identifying knowledge gaps most responsible for unexpected outcomes. The framework was used to model the effect of selenium supplementation for prostate cancer prevention and parallels the Selenium and Vitamin E Cancer Prevention Trial that showed no efficacy despite suggestive data from secondary endpoints in the Nutritional Prevention of Cancer trial and found increased incidence of high-grade prostate cancer in certain subgroups.

Conclusion

Using machine learning methods, we identified the parameters of the model that are most predictive of trial outcome and found that the top four are directly related to the rates of reactions producing methylselenol and transporting extracellular selenium into the cell as selenide. This modeling process demonstrates how the approach can be used in advance of a large clinical trial to identify the best targets for conducting further research to reduce the uncertainty in the trial outcome.

Electronic supplementary material

The online version of this article (doi:10.1186/s12918-014-0140-0) contains supplementary material, which is available to authorized users.

Value of information analysis in healthcare: a review of principles and applications

BACKGROUND

Economic evaluations are increasingly utilized to inform decisions in healthcare; however, decisions remain uncertain when they are not based on adequate evidence. Value of information (VOI) analysis has been proposed as a systematic approach to measure decision uncertainty and assess whether there is sufficient evidence to support new technologies.

SCOPE

The objective of this paper is to review the principles and applications of VOI analysis in healthcare. Relevant databases were systematically searched to identify VOI articles. The findings from the selected articles were summarized and narratively presented.

FINDINGS

Various VOI methods have been developed and applied to inform decision-making, optimally designing research studies and setting research priorities. However, the application of this approach in healthcare remains limited due to technical and policy challenges.

CONCLUSION

There is a need to create more awareness about VOI analysis, simplify its current methods, and align them with the needs of decision-making organizations.

Improving early cycle economic evaluation of diagnostic technologies

The rapidly increasing range and expense of new diagnostics, compels consideration of a different, more proactive approach to health economic evaluation of diagnostic technologies. Early cycle economic evaluation is a decision analytic approach to evaluate technologies in development so as to increase the return on investment as well as patient and societal impact. This paper describes examples of 'early cycle economic evaluations' as applied to diagnostic technologies and highlights challenges in its real-time application. It shows that especially in the field of diagnostics, with rapid technological developments and a changing regulatory climate, early cycle economic evaluation can have a guiding role to improve the efficiency of the diagnostics innovation process. In the next five years the attention will move beyond the methodological and analytic challenges of early cycle economic evaluation towards the challenge of effectively applying it to improve diagnostic research and development and patient value. Future work in this area should therefore be 'strong on principles and soft on metrics', that is, the metrics that resonate most clearly with the various decision makers in this field.

When is enough evidence enough? - Using systematic decision analysis and value-of-information analysis to determine the need for further evidence

Decision analysis (DA) and value-of-information (VOI) analysis provide a systematic, quantitative methodological framework that explicitly considers the uncertainty surrounding the currently available evidence to guide healthcare decisions. In medical decision making under uncertainty, there are two fundamental questions: 1) What decision should be made now given the best available evidence (and its uncertainty)?; 2) Subsequent to the current decision and given the magnitude of the remaining uncertainty, should we gather further evidence (i.e., perform additional studies), and if yes, which studies should be undertaken (e.g., efficacy, side effects, quality of life, costs), and what sample sizes are needed? Using the currently best available evidence, VoI analysis focuses on the likelihood of making a wrong decision if the new intervention is adopted. The value of performing further studies and gathering additional evidence is based on the extent to which the additional information will reduce this uncertainty. A quantitative framework allows for the valuation of the additional information that is generated by further research, and considers the decision maker's objectives and resource constraints. Claxton et al. summarise: "Value of information analysis can be used to inform a range of policy questions including whether a new technology should be approved based on existing evidence, whether it should be approved but additional research conducted or whether approval should be withheld until the additional evidence becomes available." [Claxton K. Value of information entry in Encyclopaedia of Health Economics, Elsevier, forthcoming 2014.] The purpose of this tutorial is to introduce the framework of systematic VoI analysis to guide further research. In our tutorial article, we explain the theoretical foundations and practical methods of decision analysis and value-of-information analysis. To illustrate, we use a simple case example of a foot ulcer (e.g., with diabetes) as well as key references from the literature, including examples for the use of the decision-analytic VoI framework by health technology assessment agencies to guide further research. These concepts may guide stakeholders involved or interested in how to determine whether or not and, if so, which additional evidence is needed to make decisions.

Value of information analysis in oncology: the value of evidence and evidence of value

PURPOSE

Value of information (VOI) analysis is a novel systematic approach for assessing whether there is sufficient evidence to support regulatory approval of new technologies, estimating the value of additional research, informing trial design, and setting research priorities. This article reviews the use of VOI methods in oncology and identifies the potential applications of VOI in this field.

METHODS

A systematic literature search was undertaken to identify studies explicitly reporting VOI analyses for interventions directed at cancer management. Articles published from 2000 onward addressing prevention, screening, diagnosis, treatment, or symptom management in oncology were selected.

RESULTS

A total of 35 articles were included in the review; most were published after 2006. The main cancers addressed were breast (n = 10; 29%), prostate (n = 5; 14%), lung (n = 5; 14%), and colorectal (n = 3; 9%). The VOI analyses were of an applied nature in 31 studies (89%). In the applied studies, VOI was used to characterize decision uncertainty in all studies and to inform future research focus in 16 (52%). Additionally, one article (3%) addressed the value of optimal trial design, and one article (3%) reported the use of VOI methods to prioritize research.

CONCLUSION

The application of VOI analysis in oncology is growing but remains limited. Benefits in oncology research and practice will potentially be optimized with an increase in the application of VOI methods to inform decision making, optimal trial design, and research prioritization in this field.