The value of information and optimal clinical trial design

Rapid Assessment of the Need for Evidence: Applying the Principles of Value of Information to Research Prioritisation

We propose a short-cut heuristic approach to rapidly estimate value of information (VOI) using information commonly reported in a research funding application to make a case for the need for further evaluative research. We develop a "Rapid VOI" approach, which focuses on uncertainty in the primary outcome of clinical effectiveness and uses this to explore the health consequences of decision uncertainty. We develop a freely accessible online tool, Rapid Assessment of the Need for Evidence (RANE), to allow for the efficient computation of the value of research. As a case study, the method was applied to a proposal for research on shoulder pain rehabilitation. The analysis was included as part of a successful application for research funding to the UK National Institute for Health and Care Research. Our approach enables research funders and applicants to rapidly estimate the value of proposed research. Rapid VOI relies on information that is readily available and reported in research funding applications. Rapid VOI supports research prioritisation and commissioning decisions where there is insufficient time and resources available to develop and validate complex decision-analytic models. The method provides a practical means for implementing VOI in practice, thus providing a starting point for deliberation and contributing to the transparency and accountability of research prioritisation decisions.

Partial-Cost Analysis and Economic Impact of Ambulatory Coronary Angioplasty in a Private Hospital in the Caribbean: Análisis Parcial de Costos e Impacto Económico de la Angioplastia Coronaria Ambulatoria en un Hospital Privado del Caribe

OBJECTIVES

This study aimed to assess direct costs of percutaneous coronary intervention (PCI) without hospital admission versus PCI with hospital admission longer than 24 hours in a private hospital-institutional perspective in the Dominican Republic in 2022.

METHODS

This study has a comparative approach based on a prospective cross-sectional partial-cost analysis. We evaluated the direct costs of 10 patients from PCI without hospital admission approach and 10 patients from a hospital admission longer than 24 hours as a control group. We used a "first-come-first-served" approach from December 2021 to March 2022. The analysis used the electronic invoice generated for each patient.

RESULTS

PCI without hospital admission approach represents $472.56 in patient savings, equivalent to a cost reduction of 12.5%. The subcosts analysis showed the pharmacy section as the main driver of the overall cost difference.

CONCLUSIONS

PCI without hospital admission was economically cost-saving compared with the control approach in direct costs in the Dominican perspective. The economic benefit is substantial and compliments the ease of use. This analysis may lead to improvements in institutional management of resources and can potentially be adapted to other health systems in the region.

Value of Information for Clinical Trial Design: The Importance of Considering All Relevant Comparators

Value of Information (VOI) analyses calculate the economic value that could be generated by obtaining further information to reduce uncertainty in a health economic decision model. VOI has been suggested as a tool for research prioritisation and trial design as it can highlight economically valuable avenues for future research. Recent methodological advances have made it increasingly feasible to use VOI in practice for research; however, there are critical differences between the VOI approach and the standard methods used to design research studies such as clinical trials. We aimed to highlight key differences between the research design approach based on VOI and standard clinical trial design methods, in particular the importance of considering the full decision context. We present two hypothetical examples to demonstrate that VOI methods are only accurate when (1) all feasible comparators are included in the decision model when designing research, and (2) all comparators are retained in the decision model once the data have been collected and a final treatment recommendation is made. Omitting comparators from either the design or analysis phase of research when using VOI methods can lead to incorrect trial designs and/or treatment recommendations. Overall, we conclude that incorrectly specifying the health economic model by ignoring potential comparators can lead to misleading VOI results and potentially waste scarce research resources.

Value of information dynamics in Disease X vaccine clinical trials

BACKGROUND

Solutions have been proposed to accelerate the development and rollout of vaccines against a hypothetical disease with epidemic or pandemic potential called Disease X. This may involve resolving uncertainties regarding the disease and the new vaccine. However the value for public health of collecting this information will depend on the time needed to perform research, but also on the time needed to produce vaccine doses. We explore this interplay, and its effect on the decision on whether or not to perform research.

METHOD

We simulate numerically the emergence and transmission of a disease in a population using a susceptible-infected-recovered (SIR) compartmental model with vaccination. Uncertainties regarding the disease and the vaccine are represented by parameter prior distributions. We vary the date at which vaccine doses are available, and the date at which information about parameters becomes available. We use the expected value of perfect information (EVPI) and the expected value of partially perfect information (EVPPI) to measure the value of information.

RESULTS

As expected, information has less or no value if it comes too late, or (equivalently) if it can only be used too late. However we also find non trivial dynamics for shorter durations of vaccine development. In this parameter area, it can be optimal to implement vaccination without waiting for information depending on the respective durations of dose production and of clinical research.

CONCLUSION

We illustrate the value of information dynamics in a Disease X outbreak scenario, and present a general approach to properly take into account uncertainties and transmission dynamics when planning clinical research in this scenario. Our method is based on numerical simulation and allows us to highlight non trivial effects that cannot otherwise be investigated.

Estimating the Potential Benefits of Confirmatory Trials for Drugs with Accelerated Approval: A Comprehensive Value of Information Framework

BACKGROUND

The US Food and Drug Administration's Accelerated Approval (AA) policy provides a pathway for patients to access potentially life-saving drugs rapidly. However, the use of surrogate endpoints, single-arm designs, and small sample sizes in preliminary trials that support AAs can lead to uncertainty regarding the clinical benefits of such drugs. This study aims to develop a comprehensive value of information (VOI) framework for assessing the potential benefits of future confirmatory trials, accounting for the various uncertainties inherent in preliminary trials.

METHODS

I formulated an expected value of information from confirmatory trial (EVICT) metric, which evaluates the potential benefits of a confirmatory trial that would reduce those uncertainties by using a clinically meaningful endpoint, a randomized control, and increased sample size. The EVICT metric can quantify the expected benefits of a well-designed confirmatory trial or an inadequately designed one that continues to use surrogate endpoints or single-arm design. The framework was illustrated using a hypothetical AA drug for metastatic breast cancer.

RESULTS

The case study demonstrates that a highly uncertain preliminary trial of an AA drug was associated with a substantial EVICT. A confirmatory trial with an increased sample size for this AA drug, utilizing a clinically meaningful endpoint and randomized control, yielded a population-level EVICT of $12.6 million. Persistently using a surrogate endpoint and single-arm trial design would reduce the EVICT by 60%.

CONCLUSIONS

This framework can provide accurate VOI estimates to guide coverage policies, value-based pricing, and the design of confirmatory trials for AA drugs.

Use of Bayesian decision analysis to maximize value in patient-centered randomized clinical trials in Parkinson's disease

A fixed one-sided significance level of 5% is commonly used to interpret the statistical significance of randomized clinical trial (RCT) outcomes. While it is necessary to reduce the false positive rate, the threshold used could be chosen quantitatively and transparently to specifically reflect patient preferences regarding benefit-risk tradeoffs as well as other considerations. How can patient preferences be explicitly incorporated into RCTs in Parkinson's disease (PD), and what is the impact on statistical thresholds for device approval? In this analysis, we apply Bayesian decision analysis (BDA) to PD patient preference scores elicited from survey data. BDA allows us to choose a sample size (n ) and significance level (α ) that maximizes the overall expected value to patients of a balanced two-arm fixed-sample RCT, where the expected value is computed under both null and alternative hypotheses. For PD patients who had previously received deep brain stimulation (DBS) treatment, the BDA-optimal significance levels fell between 4.0% and 10.0%, similar to or greater than the traditional value of 5%. Conversely, for patients who had never received DBS, the optimal significance level ranged from 0.2% to 4.4%. In both of these populations, the optimal significance level increased with the severity of the patients' cognitive and motor function symptoms. By explicitly incorporating patient preferences into clinical trial designs and the regulatory decision-making process, BDA provides a quantitative and transparent approach to combine clinical and statistical significance. For PD patients who have never received DBS treatment, a 5% significance threshold may not be conservative enough to reflect their risk-aversion level. However, this study shows that patients who previously received DBS treatment present a higher tolerance to accept therapeutic risks in exchange for improved efficacy which is reflected in a higher statistical threshold.

Using early health economic modeling to inform medical innovation development: a soft robotic sock in poststroke patients in Singapore

OBJECTIVES

Based on a real-world collaboration with innovators in applying early health economic modeling, we aimed to offer practical steps that health technology assessment (HTA) researchers and innovators can follow and promote the usage of early HTA among research and development (R&D) communities.

METHODS

The HTA researcher was approached by the innovator to carry out an early HTA ahead of the first clinical trial of the technology, a soft robotic sock for poststroke patients. Early health economic modeling was selected to understand the potential value of the technology and to help uncover the information gap. Threshold analysis was used to identify the target product profiles. Value-of-information analysis was conducted to understand the uncertainties and the need for further research.

RESULTS

Based on the expected price and clinical effectiveness by the innovator, the new technology was found to be cost-saving compared to the current practice. Risk reduction in deep vein thrombosis and ankle contracture, the incidence rate of ankle contracture, the compliance rate of the new technology, and utility scores were found to have high impacts on the value-for-money of the new technology. The value of information was low if the new technology can achieve the expected clinical effectiveness. A list of parameters was recommended for data collection in the impending clinical trial.

CONCLUSIONS

This work, based on a real-world collaboration, has illustrated that early health economic modeling can inform medical innovation development. We provided practical steps in order to achieve more efficient R&D investment in medical innovation moving forward.

Predictive validity in drug discovery: what it is, why it matters and how to improve it

Successful drug discovery is like finding oases of safety and efficacy in chemical and biological deserts. Screens in disease models, and other decision tools used in drug research and development (R&D), point towards oases when they score therapeutic candidates in a way that correlates with clinical utility in humans. Otherwise, they probably lead in the wrong direction. This line of thought can be quantified by using decision theory, in which 'predictive validity' is the correlation coefficient between the output of a decision tool and clinical utility across therapeutic candidates. Analyses based on this approach reveal that the detectability of good candidates is extremely sensitive to predictive validity, because the deserts are big and oases small. Both history and decision theory suggest that predictive validity is under-managed in drug R&D, not least because it is so hard to measure before projects succeed or fail later in the process. This article explains the influence of predictive validity on R&D productivity and discusses methods to evaluate and improve it, with the aim of supporting the application of more effective decision tools and catalysing investment in their creation.

A Conceptual Framework for Life-Cycle Health Technology Assessment

OBJECTIVES

Health technology assessment (HTA) uses evidence appraisal and synthesis with economic evaluation to inform adoption decisions. Standard HTA processes sometimes struggle to (1) support decisions that involve significant uncertainty and (2) encourage continued generation of and adaptation to new evidence. We propose the life-cycle (LC)-HTA framework, addressing these challenges by providing additional tools to decision makers and improving outcomes for all stakeholders.

METHODS

Under the LC-HTA framework, HTA processes align to LC management. LC-HTA introduces changes in HTA methods to minimize analytic time while optimizing decision certainty. Where decision uncertainty exists, we recommend risk-based pricing and research-oriented managed access (ROMA). Contractual procurement agreements define the terms of reassessment and provide additional decision options to HTA agencies. LC-HTA extends value-of-information methods to inform ROMA agreements, leveraging routine, administrative data, and registries to reduce uncertainty.

RESULTS

LC-HTA enables the adoption of high-value high-risk innovations while improving health system sustainability through risk-sharing and reducing uncertainty. Responsiveness to evolving evidence is improved through contractually embedded decision rules to simplify reassessment. ROMA allows conditional adoption to obtain additional information, with confidence that the net value of that adoption decision is positive.

CONCLUSIONS

The LC-HTA framework improves outcomes for patients, sponsors, and payers. Patients benefit through earlier access to new technologies. Payers increase the value of the technologies they invest in and gain mechanisms to review investments. Sponsors benefit through greater certainty in outcomes related to their investment, swifter access to markets, and greater opportunities to demonstrate value.

The Value of the Information That Can Be Generated: Optimizing Study Design to Enable the Study of Treatments Addressing an Unmet Need for Rare Pathogens

In traditional phase 3 trials confirming safety and efficacy of new treatments relative to a comparator, a one-sided type I error rate of 2.5% is traditionally used, and typically leads to minimum sizes of 300-600 subjects per study. However, for rare pathogens, it may be necessary to work with data from as few as 50–100 subjects. For areas with a high unmet need, there is a balance between traditional type I error and power and enabling feasible studies. In such cases, an alternative one-sided alpha level of 5% or 10% should be considered and we review herein the implications of such approaches. Resolving this question requires engagement of patients, the medical community, regulatory agencies, and trial sponsors.

Image-guided navigation for locally advanced primary and locally recurrent rectal cancer: evaluation of its early cost-effectiveness

BACKGROUND

A first pilot study showed that an image-guided navigation system could improve resection margin rates in locally advanced (LARC) and locally recurrent rectal cancer (LRRC) patients. Incremental surgical innovation is often implemented without reimbursement consequences, health economic aspects should however also be taken into account. This study evaluates the early cost-effectiveness of navigated surgery compared to standard surgery in LARC and LRRC.

METHODS

A Markov decision model was constructed to estimate the expected costs and outcomes for navigated and standard surgery. The input parameters were based on pilot data from a prospective (navigation cohort n = 33) and retrospective (control group n = 142) data. Utility values were measured in a comparable group (n = 63) through the EQ5D-5L. Additionally, sensitivity and value of information analyses were performed.

RESULTS

Based on this early evaluation, navigated surgery showed incremental costs of €3141 and €2896 in LARC and LRRC. In LARC, navigated surgery resulted in 2.05 Quality-Adjusted Life Years (QALYs) vs 2.02 QALYs for standard surgery. For LRRC, we found 1.73 vs 1.67 QALYs respectively. This showed an Incremental Cost-Effectiveness Ratio (ICER) of €136.604 for LARC and €52.510 for LRRC per QALY gained. In scenario analyses, optimal utilization rates of the navigation technology lowered the ICER to €61.817 and €21.334 for LARC and LRRC. The ICERs of both indications were most sensitive to uncertainty surrounding the risk of progression in the first year after surgery, the risk of having a positive surgical margin, and the costs of the navigation system.

CONCLUSION

Adding navigation system use is expected to be cost-effective in LRRC and has the potential to become cost-effective in LARC. To increase the probability of being cost-effective, it is crucial to optimize efficient use of both the hybrid OR and the navigation system and identify subgroups where navigation is expected to show higher effectiveness.

Economic evaluations for intensive care unit randomised clinical trials in Australia and New Zealand: Practical recommendations for researchers

OBJECTIVES

Economic evaluations of intensive care unit (ICU) interventions have specific considerations, including how to cost ICU stays and accurately measure quality of life in survivors. The aim of this article was to develop best practice recommendations for economic evaluations alongside future ICU randomised controlled trials (RCTs).

REVIEW METHODS

We collated our experience based on expert consensus across several recent economic evaluations to provide best-practice, practical recommendations for researchers conducting economic evaluations alongside RCTs in the ICU. Recommendations were structured according to the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) Consolidated Health Economic Evaluation Reporting Standards (CHEERS) Task Force Report.

RESULTS

We discuss recommendations across the components of economic evaluations, including: types of economic evaluation, the population and sample size, study perspective, comparators, time horizon, choice of health outcomes, measurement of effectiveness, measurement and valuation of quality of life, estimating resources and costs, analytical methods, and the increment cost-effectiveness ratio. We also provide future directions for research with regard to developing more robust economic evaluations for the ICU.

CONCLUSION

Economic evaluations should be built alongside ICU RCTs and should be designed a priori using appropriate follow-up and data collection to capture patient-relevant outcomes. Further work is needed to improve the quality of data available for linkage in Australia as well as developing costing methods for the ICU and appropriate quality of life measurements.

Value of Information Analysis in Models to Inform Health Policy

Value of information (VoI) is a decision-theoretic approach to estimating the expected benefits from collecting further information of different kinds, in scientific problems based on combining one or more sources of data. VoI methods can assess the sensitivity of models to different sources of uncertainty and help to set priorities for further data collection. They have been widely applied in healthcare policy making, but the ideas are general to a range of evidence synthesis and decision problems. This article gives a broad overview of VoI methods, explaining the principles behind them, the range of problems that can be tackled with them, and how they can be implemented, and discusses the ongoing challenges in the area.

Modelling approaches for histology-independent cancer drugs to inform NICE appraisals: a systematic review and decision-framework

BACKGROUND

The first histology-independent marketing authorisation in Europe was granted in 2019. This was the first time that a cancer treatment was approved based on a common biomarker rather than the location in the body at which the tumour originated. This research aims to explore the implications for National Institute for Health and Care Excellence appraisals.

METHODS

Targeted reviews were undertaken to determine the type of evidence that is likely to be available at the point of marketing authorisation and the analyses required to support National Institute for Health and Care Excellence appraisals. Several challenges were identified concerning the design and conduct of trials for histology-independent products, the greater levels of heterogeneity within the licensed population and the use of surrogate end points. We identified approaches to address these challenges by reviewing key statistical literature that focuses on the design and analysis of histology-independent trials and by undertaking a systematic review to evaluate the use of response end points as surrogate outcomes for survival end points. We developed a decision framework to help to inform approval and research policies for histology-independent products. The framework explored the uncertainties and risks associated with different approval policies, including the role of further data collection, pricing schemes and stratified decision-making.

RESULTS

We found that the potential for heterogeneity in treatment effects, across tumour types or other characteristics, is likely to be a central issue for National Institute for Health and Care Excellence appraisals. Bayesian hierarchical methods may serve as a useful vehicle to assess the level of heterogeneity across tumours and to estimate the pooled treatment effects for each tumour, which can inform whether or not the assumption of homogeneity is reasonable. Our review suggests that response end points may not be reliable surrogates for survival end points. However, a surrogate-based modelling approach, which captures all relevant uncertainty, may be preferable to the use of immature survival data. Several additional sources of heterogeneity were identified as presenting potential challenges to National Institute for Health and Care Excellence appraisal, including the cost of testing, baseline risk, quality of life and routine management costs. We concluded that a range of alternative approaches will be required to address different sources of heterogeneity to support National Institute for Health and Care Excellence appraisals. An exemplar case study was developed to illustrate the nature of the assessments that may be required.

CONCLUSIONS

Adequately designed and analysed basket studies that assess the homogeneity of outcomes and allow borrowing of information across baskets, where appropriate, are recommended. Where there is evidence of heterogeneity in treatment effects and estimates of cost-effectiveness, consideration should be given to optimised recommendations. Routine presentation of the scale of the consequences of heterogeneity and decision uncertainty may provide an important additional approach to the assessments specified in the current National Institute for Health and Care Excellence methods guide.

FURTHER RESEARCH

Further exploration of Bayesian hierarchical methods could help to inform decision-makers on whether or not there is sufficient evidence of homogeneity to support pooled analyses. Further research is also required to determine the appropriate basis for apportioning genomic testing costs where there are multiple targets and to address the challenges of uncontrolled Phase II studies, including the role and use of surrogate end points.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Evidence Synthesis programme and will be published in full in Health Technology Assessment; Vol. 25, No. 76. See the NIHR Journals Library website for further project information.

The Optimal Sample Size for Usability Testing, From the Manufacturer's Perspective: A Value-of-Information Approach

OBJECTIVES

For medical devices, a usability assessment is mandatory for market access; the objective is to detect potentially harmful use errors that stem from the device's design. The manufacturer assesses the final version of the device and determines the risk-benefit ratio for remaining errors. Nevertheless, the decision rule currently used to determine the sample size for this testing has statistical limitations and the lack of a clear decision-making perspective.

METHODS

As an alternative, we developed a value-of-information analysis from the medical device manufacturer's perspective. The consequences of use errors not detected during usability testing and the errors' probability of occurrence were embedded in a loss function. The value of further testing was assessed as a reduction in the expected loss for the manufacturer. The optimal sample size was determined using the expected net benefit of sampling (ENBS) (the difference between the value provided by new participants and the cost of their inclusion).

RESULTS

The value-of-information approach was applied to a real usability test of a needle-free adrenaline autoinjector. The initial estimate (performed on the first n = 20 participants) gave an optimal sample size of 100 participants and an ENBS of €255 453. This estimation was updated iteratively as new participants were included. After the inclusion of 90 participants, the ENBS was null for any sample size; hence, the cost of adding more participants outweighed the expected value of information, and therefore, the study could be stopped.

CONCLUSIONS

On the basis of these results, our method seems to be highly suitable for sample size estimation in the usability testing of medical devices before market access.

Endovascular stent grafting and open surgical replacement for chronic thoracic aortic aneurysms: a systematic review and prospective cohort study

Background

The management of chronic thoracic aortic aneurysms includes conservative management, watchful waiting, endovascular stent grafting and open surgical replacement. The Effective Treatments for Thoracic Aortic Aneurysms (ETTAA) study investigates timing and intervention choice.

Objective

To describe pre- and post-intervention management of and outcomes for chronic thoracic aortic aneurysms.

Design

A systematic review of intervention effects; a Delphi study of 360 case scenarios based on aneurysm size, location, age, operative risk and connective tissue disorders; and a prospective cohort study of growth, clinical outcomes, costs and quality of life.

Setting

Thirty NHS vascular/cardiothoracic units.

Participants

Patients aged > 17 years who had existing or new aneurysms of ≥ 4 cm in diameter in the arch, descending or thoracoabdominal aorta.

Interventions

Endovascular stent grafting and open surgical replacement.

Main outcomes

Pre-intervention aneurysm growth, pre-/post-intervention survival, clinical events, readmissions and quality of life; and descriptive statistics for costs and quality-adjusted life-years over 12 months and value of information using a propensity score-matched subsample.

Results

The review identified five comparative cohort studies (endovascular stent grafting patients, n = 3955; open surgical replacement patients, n = 21,197). Pooled short-term all-cause mortality favoured endovascular stent grafting (odds ratio 0.71, 95% confidence interval 0.51 to 0.98; no heterogeneity). Data on survival beyond 30 days were mixed. Fewer short-term complications were reported with endovascular stent grafting. The Delphi study included 20 experts (13 centres). For patients with aneurysms of ≤ 6.0 cm in diameter, watchful waiting was preferred. For patients with aneurysms of > 6.0 cm, open surgical replacement was preferred in the arch, except for elderly or high-risk patients, and in the descending aorta if patients had connective tissue disorders. Otherwise endovascular stent grafting was preferred. Between 2014 and 2018, 886 patients were recruited (watchful waiting, n = 489; conservative management, n = 112; endovascular stent grafting, n = 150; open surgical replacement, n = 135). Pre-intervention death rate was 8.6% per patient-year; 49.6% of deaths were aneurysm related. Death rates were higher for women (hazard ratio 1.79, 95% confidence interval 1.25 to 2.57; p = 0.001) and older patients (age 61–70 years: hazard ratio 2.50, 95% confidence interval 0.76 to 5.43; age 71–80 years: hazard ratio 3.49, 95% confidence interval 1.26 to 9.66; age > 80 years: hazard ratio 7.01, 95% confidence interval 2.50 to 19.62; all compared with age < 60 years, p < 0.001) and per 1-cm increase in diameter (hazard ratio 1.90, 95% confidence interval 1.65 to 2.18; p = 0.001). The results were similar for aneurysm-related deaths. Decline per year in quality of life was greater for older patients (additional change –0.013 per decade increase in age, 95% confidence interval –0.019 to –0.007; p < 0.001) and smokers (additional change for ex-smokers compared with non-smokers 0.003, 95% confidence interval –0.026 to 0.032; additional change for current smokers compared with non-smokers –0.034, 95% confidence interval –0.057 to –0.01; p = 0.004). At the time of intervention, endovascular stent grafting patients were older (age difference 7.1 years; 95% confidence interval 4.7 to 9.5 years; p < 0.001) and more likely to be smokers (75.8% vs. 66.4%; p = 0.080), have valve disease (89.9% vs. 71.6%; p < 0.0001), have chronic obstructive pulmonary disease (21.3% vs. 13.3%; p = 0.087), be at New York Heart Association stage III/IV (22.3% vs. 16.0%; p = 0.217), have lower levels of haemoglobin (difference –6.8 g/l, 95% confidence interval –11.2 to –2.4 g/l; p = 0.003) and take statins (69.3% vs. 42.2%; p < 0.0001). Ten (6.7%) endovascular stent grafting and 15 (11.1%) open surgical replacement patients died within 30 days of the procedure (p = 0.2107). One-year overall survival was 82.5% (95% confidence interval 75.2% to 87.8%) after endovascular stent grafting and 79.3% (95% confidence interval 71.1% to 85.4%) after open surgical replacement. Variables affecting survival were aneurysm site, age, New York Heart Association stage and time waiting for procedure. For endovascular stent grafting, utility decreased slightly, by –0.017 (95% confidence interval –0.062 to 0.027), in the first 6 weeks. For open surgical replacement, there was a substantial decrease of –0.160 (95% confidence interval –0.199 to –0.121; p < 0.001) up to 6 weeks after the procedure. Over 12 months endovascular stent grafting was less costly, with higher quality-adjusted life-years. Formal economic analysis was unfeasible.

Limitations

The study was limited by small numbers of patients receiving interventions and because only 53% of patients were suitable for both interventions.

Conclusions

Small (4–6 cm) aneurysms require close observation. Larger (> 6 cm) aneurysms require intervention without delay. Endovascular stent grafting and open surgical replacement were successful for carefully selected patients, but cost comparisons were unfeasible. The choice of intervention is well established, but the timing of intervention remains challenging.

Future work

Further research should include an analysis of the risk factors for growth/rupture and long-term outcomes.

Trial registration

Current Controlled Trials ISRCTN04044627 and NCT02010892.

Funding

This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 26, No. 6. See the NIHR Journals Library website for further project information.

Using health technology assessment to set priority, inform target product profiles, and design clinical study for health innovation

Early health technology assessment (early HTA) is a useful tool in guiding the innovation development process in medical technology development. However, the application of early HTA is sub-optimal amongst research and development (R&D) communities due to several challenges. In this paper, we presented a case study of application of early HTA by drawing on the experience from a workshop conducted for the Singapore government's medical technology innovation agency. The framework developed can help maximise the chance of the newly developed technology being accepted and widely used. By providing step-by-step guidance, this work aims to translate early HTA into a practical tool and promote the application of early HTA amongst R&D communities.

Prioritisation and design of clinical trials

Clinical trials require participation of numerous patients, enormous research resources and substantial public funding. Time-consuming trials lead to delayed implementation of beneficial interventions and to reduced benefit to patients. This manuscript discusses two methods for the allocation of research resources and reviews a framework for prioritisation and design of clinical trials. The traditional error-driven approach of clinical trial design controls for type I and II errors. However, controlling for those statistical errors has limited relevance to policy makers. Therefore, this error-driven approach can be inefficient, waste research resources and lead to research with limited impact on daily practice. The novel value-driven approach assesses the currently available evidence and focuses on designing clinical trials that directly inform policy and treatment decisions. Estimating the net value of collecting further information, prior to undertaking a trial, informs a decision maker whether a clinical or health policy decision can be made with current information or if collection of extra evidence is justified. Additionally, estimating the net value of new information guides study design, data collection choices, and sample size estimation. The value-driven approach ensures the efficient use of research resources, reduces unnecessary burden to trial participants, and accelerates implementation of beneficial healthcare interventions.

Combining Model-Based Clinical Trial Simulation, Pharmacoeconomics, and Value of Information to Optimize Trial Design

The Bayesian decision‐analytic approach to trial design uses prior distributions for treatment effects, updated with likelihoods for proposed trial data. Prior distributions for treatment effects based on previous trial results risks sample selection bias and difficulties when a proposed trial differs in terms of patient characteristics, medication adherence, or treatment doses and regimens. The aim of this study was to demonstrate the utility of using pharmacometric‐based clinical trial simulation (CTS) to generate prior distributions for use in Bayesian decision‐theoretic trial design. The methods consisted of four principal stages: a CTS to predict the distribution of treatment response for a range of trial designs; Bayesian updating for a proposed sample size; a pharmacoeconomic model to represent the perspective of a reimbursement authority in which price is contingent on trial outcome; and a model of the pharmaceutical company return on investment linking drug prices to sales revenue. We used a case study of febuxostat versus allopurinol for the treatment of hyperuricemia in patients with gout. Trial design scenarios studied included alternative treatment doses, inclusion criteria, input uncertainty, and sample size. Optimal trial sample sizes varied depending on the uncertainty of model inputs, trial inclusion criteria, and treatment doses. This interdisciplinary framework for trial design and sample size calculation may have value in supporting decisions during later phases of drug development and in identifying costly sources of uncertainty, and thus inform future research and development strategies.

Informed and uninformed empirical therapy policies

We argue that a proper distinction must be made between informed and uninformed decision making when setting empirical therapy policies, as this allows one to estimate the value of gathering more information about the pathogens and their transmission and thus to set research priorities. We rely on the stochastic version of a compartmental model to describe the spread of an infecting organism in a health care facility and the emergence and spread of resistance to two drugs. We focus on information and uncertainty regarding the parameters of this model. We consider a family of adaptive empirical therapy policies. In the uninformed setting, the best adaptive policy allowsone to reduce the average cumulative infected patient days over 2 years by 39.3% (95% confidence interval (CI), 30.3-48.1%) compared to the combination therapy. Choosing empirical therapy policies while knowing the exact parameter values allows one to further decrease the cumulative infected patient days by 3.9% (95% CI, 2.1-5.8%) on average. In our setting, the benefit of perfect information might be offset by increased drug consumption.

Competitive Swarm Optimizer with Mutated Agents for Finding Optimal Designs for Nonlinear Regression Models with Multiple Interacting Factors

This paper proposes a novel enhancement for Competitive Swarm Optimizer (CSO) by mutating loser particles (agents) from the swarm to increase the swarm diversity and improve space exploration capability, namely Competitive Swarm Optimizer with Mutated Agents (CSO-MA). The selection mechanism is carried out so that it does not retard the search if agents are exploring in promising areas. Simulation results show that CSO-MA has a better exploration-exploitation balance than CSO and generally outperforms CSO, which is one of the state-of-the-art metaheuristic algorithms for optimization. We show additionally that it also generally outperforms swarm based types of algorithms and an exemplary and popular non-swarm based algorithm called Cuckoo search, without requiring a lot more CPU time. We apply CSO-MA to find a c-optimal approximate design for a high-dimensional optimal design problem when other swarm algorithms were not able to. As applications, we use the CSO-MA to search various optimal designs for a series of high-dimensional statistical models. The proposed CSO-MA algorithm is a general-purpose optimizing tool and can be directly amended to find other types of optimal designs for nonlinear models, including optimal exact designs under a convex or non-convex criterion.

Value of Information Analytical Methods: Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force

The allocation of healthcare resources among competing priorities requires an assessment of the expected costs and health effects of investing resources in the activities and of the opportunity cost of the expenditure. To date, much effort has been devoted to assessing the expected costs and health effects, but there remains an important need to also reflect the consequences of uncertainty in resource allocation decisions and the value of further research to reduce uncertainty. Decision making with uncertainty may turn out to be suboptimal, resulting in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to better inform resource decisions. This value can be quantified using value of information (VOI) analysis. This report from the ISPOR VOI Task Force describes methods for computing 4 VOI measures: the expected value of perfect information, expected value of partial perfect information (EVPPI), expected value of sample information (EVSI), and expected net benefit of sampling (ENBS). Several methods exist for computing EVPPI and EVSI, and this report provides guidance on selecting the most appropriate method based on the features of the decision problem. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing VOI analyses. The software needed to compute VOI is discussed, and areas for future research are highlighted.

Value of Information Analysis for Research Decisions-An Introduction: Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force

Healthcare resource allocation decisions made under conditions of uncertainty may turn out to be suboptimal. In a resource constrained system in which there is a fixed budget, these suboptimal decisions will result in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to make better resource allocation decisions. This value can be quantified using a value of information (VOI) analysis. This report, from the ISPOR VOI Task Force, introduces VOI analysis, defines key concepts and terminology, and outlines the role of VOI for supporting decision making, including the steps involved in undertaking and interpreting VOI analyses. The report is specifically aimed at those tasked with making decisions about the adoption of healthcare or the funding of healthcare research. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing the results of VOI analyses.

A systematic review of health economic evaluations of proton beam therapy for adult cancer: Appraising methodology and quality

BACKGROUND AND PURPOSE

With high treatment costs and limited capacity, decisions on which adult patients to treat with proton beam therapy (PBT) must be based on the relative value compared to the current standard of care. Cost-utility analyses (CUAs) are the gold-standard method for doing this. We aimed to appraise the methodology and quality of CUAs in this area.

MATERIALS AND METHODS

We performed a systematic review of the literature to identify CUA studies of PBT in adult disease using MEDLINE, EMBASE, EconLIT, NHS Economic Evaluation Database (NHS EED), Web of Science, and the Tufts Medical Center Cost-Effectiveness Analysis Registry from 1st January 2010 up to 6th June 2018. General characteristics, information relating to modelling approaches, and methodological quality were extracted and synthesized narratively.

RESULTS

Seven PBT CUA studies in adult disease were identified. Without randomised controlled trials to inform the comparative effectiveness of PBT, studies used either results from one-armed studies, or dose-response models derived from radiobiological and epidemiological studies of PBT. Costing methods varied widely. The assessment of model quality highlighted a lack of transparency in the identification of model parameters, and absence of external validation of model outcomes. Furthermore, appropriate assessment of uncertainty was often deficient.

CONCLUSION

In order to foster credibility, future CUA studies must be more systematic in their approach to evidence synthesis and expansive in their consideration of uncertainties in light of the lack of clinical evidence.

The Curve of Optimal Sample Size (COSS): A Graphical Representation of the Optimal Sample Size from a Value of Information Analysis

Value of information (VOI) analysis quantifies the opportunity cost associated with decision uncertainty, and thus informs the value of collecting further information to avoid this cost. VOI can inform study design, optimal sample size selection, and research prioritization. Recent methodological advances have reduced the computational burden of conducting VOI analysis and have made it easier to evaluate the expected value of sample information, the expected net benefit of sampling, and the optimal sample size of a study design ([Formula: see text]). The volume of VOI analyses being published is increasing, and there is now a need for VOI studies to conduct sensitivity analyses on VOI-specific parameters. In this practical application, we introduce the curve of optimal sample size (COSS), which is a graphical representation of [Formula: see text] over a range of willingness-to-pay thresholds and VOI parameters (example data and R code are provided). In a single figure, the COSS presents summary data for decision makers to determine the sample size that optimizes research funding given their operating characteristics. The COSS also presents variation in the optimal sample size given variability or uncertainty in VOI parameters. The COSS represents an efficient and additional approach for summarizing results from a VOI analysis.

Estimating the Expected Value of Sample Information across Different Sample Sizes Using Moment Matching and Nonlinear Regression

Background. The expected value of sample information (EVSI) determines the economic value of any future study with a specific design aimed at reducing uncertainty about the parameters underlying a health economic model. This has potential as a tool for trial design; the cost and value of different designs could be compared to find the trial with the greatest net benefit. However, despite recent developments, EVSI analysis can be slow, especially when optimizing over a large number of different designs. Methods. This article develops a method to reduce the computation time required to calculate the EVSI across different sample sizes. Our method extends the moment-matching approach to EVSI estimation to optimize over different sample sizes for the underlying trial while retaining a similar computational cost to a single EVSI estimate. This extension calculates the posterior variance of the net monetary benefit across alternative sample sizes and then uses Bayesian nonlinear regression to estimate the EVSI across these sample sizes. Results. A health economic model developed to assess the cost-effectiveness of interventions for chronic pain demonstrates that this EVSI calculation method is fast and accurate for realistic models. This example also highlights how different trial designs can be compared using the EVSI. Conclusion. The proposed estimation method is fast and accurate when calculating the EVSI across different sample sizes. This will allow researchers to realize the potential of using the EVSI to determine an economically optimal trial design for reducing uncertainty in health economic models. Limitations. Our method involves rerunning the health economic model, which can be more computationally expensive than some recent alternatives, especially in complex models.

A cost effectiveness analysis of maintenance cognitive stimulation therapy (MCST) for people with dementia: examining the influence of cognitive ability and living arrangements

OBJECTIVES

Identify if cost-effectiveness of Maintenance Cognitive Simulation Therapy (MCST) differs by type of living arrangement and cognitive ability of the person with dementia. Next, a value of information analysis is performed to inform decisions about future research.

METHODS

Incremental cost-effectiveness analysis applying seemingly unrelated regressions using data from a multicentre RCT of MCST versus treatment as usual in a population which had already received 7 weeks of CST for dementia (ISRCTN: 26286067). The findings from the cost-effectiveness analysis are used to inform a value of information analysis.

RESULTS

The results are dependent upon how quality adjusted life years (QALYs) are measured. MCST might be cost-effective compared to standard treatment for those who live alone and those with higher levels of cognitive functioning. If a further RCT was to be conducted for this sub-group of the population, value of information analysis suggests a total sample of 48 complete cases for both sub-groups would be required for a two-arm trial. The expected net gain of conducting this future research is £920 million.

CONCLUSION

Preliminary results suggest that MCST may be most cost-efficient for people with dementia who live alone and/or who have higher cognition. Future research in this area is needed.

Value of Information: Sensitivity Analysis and Research Design in Bayesian Evidence Synthesis

Suppose we have a Bayesian model that combines evidence from several different sources. We want to know which model parameters most affect the estimate or decision from the model, or which of the parameter uncertainties drive the decision uncertainty. Furthermore, we want to prioritize what further data should be collected. These questions can be addressed by Value of Information (VoI) analysis, in which we estimate expected reductions in loss from learning specific parameters or collecting data of a given design. We describe the theory and practice of VoI for Bayesian evidence synthesis, using and extending ideas from health economics, computer modeling and Bayesian design. The methods are general to a range of decision problems including point estimation and choices between discrete actions. We apply them to a model for estimating prevalence of HIV infection, combining indirect information from surveys, registers, and expert beliefs. This analysis shows which parameters contribute most of the uncertainty about each prevalence estimate, and the expected improvements in precision from specific amounts of additional data. These benefits can be traded with the costs of sampling to determine an optimal sample size. Supplementary materials for this article, including a standardized description of the materials available for reproducing the work, are available as an online supplement.

A new approach for sample size calculation in cost-effectiveness studies based on value of information

Background

Value of information is now recognized as a reference method in the decision process underpinning cost-effectiveness evaluation. The expected value of perfect information (EVPI) is the expected value from completely reducing the uncertainty surrounding the cost-effectiveness of an innovative intervention.

Among sample size calculation methods used in cost-effectiveness studies, only one is coherent with this decision framework. It uses a Bayesian approach and requires data of a pre-existing cost-effectiveness study to derive a valid prior EVPI. When evaluating the cost-effectiveness of innovations, no observed prior EVPI is usually available to calculate the sample size.

We here propose a sample size calculation method for cost-effectiveness studies, that follows the value of information theory, and, being frequentist, can be based on assumptions if no observed prior EVPI is available.

Methods

The general principle of our method is to define the sampling distribution of the incremental net monetary benefit (ΔB), or the distribution of ΔB that would be observed in a planned cost-effectiveness study of size n. Based on this sampling distribution, the EVPI that would remain at the end of the trial (EVPI_n) is estimated. The optimal sample size of the planned cost-effectiveness study is the n for which the cost of including an additional participant becomes equal or higher than the value of the information gathered through this inclusion.

Results

Our method is illustrated through four examples. The first one is used to present the method in depth and describe how the sample size may vary according to the parameters’ value. The three other examples are used to illustrate in different situations how the sample size may vary according to the ceiling cost-effectiveness ratio, and how it compares with a test statistic-based method. We developed an R package (EBASS) to run these calculations.

Conclusions

Our sample size calculation method follows the value of information theory that is now recommended for analyzing and interpreting cost-effectiveness data, and sets the size of a study that balances its cost and the value of its information.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0571-1) contains supplementary material, which is available to authorized users.

Early cost-effectiveness of tumor infiltrating lymphocytes (TIL) for second line treatment in advanced melanoma: a model-based economic evaluation

Background

An emerging immunotherapy is infusion of tumor infiltrating Lymphocytes (TIL), with objective response rates of around 50% versus 19% for ipilimumab. As an Advanced Therapeutic Medicinal Products (ATMP), TIL is highly personalized and complex therapy. It requests substantial upfront investments from the hospital in: expensive lab-equipment, staff expertise and training, as well as extremely tight hospital logistics. Therefore, an early health economic modelling study, as part of a Coverage with Evidence Development (CED) program, was performed.

Methods

We used a Markov decision model to estimate the expected costs and outcomes (quality-adjusted life years; QALYs) for TIL versus ipilimumab for second line treatment in metastatic melanoma patients from a Dutch health care perspective over a life long time horizon. Three mutually exclusive health states (stable disease (responders)), progressive disease and death) were modelled. To inform further research prioritization, Value of Information (VOI) analysis was performed.

Results

TIL is expected to generate more QALYs compared to ipilimumab (0.45 versus 0.38 respectively) at lower incremental cost (presently €81,140 versus €94,705 respectively) resulting in a dominant ICER (less costly and more effective). Based on current information TIL is dominating ipilimumab and has a probability of 86% for being cost effective at a cost/QALY threshold of €80,000. The Expected Value of Perfect Information (EVPI) amounted to €3 M.

Conclusions

TIL is expected to have the highest probability of being cost-effective in second line treatment for advanced melanoma compared to ipilimumab. To reduce decision uncertainty, a clinical trial investigating e.g. costs and survival seems most valuable. This is currently being undertaken as part of a CED program in the Netherlands Cancer Institute, Amsterdam, the Netherlands, in collaboration with Denmark.

Using value of information methods to determine the optimal sample size for effectiveness trials of alcohol interventions for HIV-infected patients in East Africa

Background

Unhealthy alcohol consumption exacerbates the HIV epidemic in East Africa. Potential benefits of new trials that test the effectiveness of alcohol interventions could not be evaluated by traditional sampling methods. Given the competition for health care resources in East Africa, this study aims to determine the optimal sample size given the opportunity cost of potentially re-allocating trial funds towards cost-effective alcohol treatments.

Methods

We used value of information methods to determine the optimal sample size by maximizing the expected net benefit of sampling for a hypothetical 2-arm intervention vs. control randomized trial, across ranges of policymaker’s willingness-to-pay for the health benefit of an intervention. Probability distributions describing the relative likelihood of alternative trial results were imputed based on prior studies. In the base case, policymaker’s willingness-to-pay was based on a simultaneously resource-constrained priority (routine HIV virological testing). Sensitivity analysis was performed for various willingness-to-pay thresholds and intervention durations.

Results

A new effectiveness trial accounting for the benefit of more precise decision-making on alcohol intervention implementation would benefit East Africa $67,000 with the optimal sample size of 100 persons per arm under the base case willingness-to-pay threshold and intervention duration of 20 years. At both a conservative willingness-to-pay of 1 x GDP/capita and a high willingness-to-pay of 3 x GDP/capita for an additional health gain added by an alcohol intervention, a new trial was not recommended due to limited decision uncertainty. When intervention duration was 10 or 5 years, there was no return on investment across suggested willingness-to-pay thresholds.

Conclusions

Value of information methods could be used as an alternative approach to assist the efficient design of alcohol trials. If reducing unhealthy alcohol use is a long-term goal for HIV programs in East Africa, additional new trials with optimal sample sizes ranging from 100 to 250 persons per arm could save the opportunity cost of implementing less cost-effective alcohol strategies in HIV prevention. Otherwise, conducting a new trial is not recommended.

Decision Criterion and Value of Information Analysis: Optimal Aspirin Dosage for Secondary Prevention of Cardiovascular Events

BACKGROUND

In value of information (VOI) calculations, such as the expected value of perfect information (EVPI), partial perfect information (EVPPI), sample information (EVSI) or implementation (EVIM), the maximum expected value criterion defines the decision making criterion for the adoption of decisions for treatments. However, because decision makers are often risk averse, the uncertainty that accompanies a decision problem may influence adoption decisions.

METHODS

VOI estimates were studied based on 2 alternate decision making criteria: 1) maximum expected value and 2) 95% credible intervals. These criteria were applied to a probabilistic minimal lifetime model of incident cardiovascular events and mortality among target patients comparing 2 daily doses of aspirin (81 mg and 325 mg). Model parameters were based on literature reviews and data analyses.

RESULTS

Expected life-years under 81 v. 325 mg of aspirin were estimated to be 14.86 (SE, 0.10) and 14.72 (0.31) respectively, with a difference of 0.14 (0.29). The probability that 81 mg was optimal was estimated to be 67%. Under Decision Criterion 1, EVIM and EVPI were about 233-thousand and 411-thousand years, respectively. Under Criterion 2, EVIM was undefined, as there remains ambiguity about what to implement. Consequently, EVPI becomes the entire 644-thousand years. Also, under Criterion 1, EVSI reaches an asymptote at a sample size of 10,000 per arm, with minimal gains in value beyond a 5,000 person per arm trial. With Criterion 2, a sample size of 10,000 per arm or higher is substantially more valuable than lower sample sizes.

CONCLUSION

Alternative decision criteria for treatment adoption change the VOI. Decision criteria should be justified for VOI analyses. If multiple criteria may be relevant, analysts should complete VOI estimates using multiple criteria.

Invasive urodynamic testing prior to surgical treatment for stress urinary incontinence in women: cost-effectiveness and value of information analyses in the context of a mixed methods feasibility study

BACKGROUND

INVESTIGATE-I (INVasive Evaluation before Surgical Treatment of Incontinence Gives Added Therapeutic Effect?) was a mixed methods study to assess the feasibility of a future randomised controlled trial of invasive urodynamic testing (IUT) prior to surgery for stress urinary incontinence (SUI) in women. Here we report one of the study's five components, with the specific objectives of (i) exploring the cost-effectiveness of IUT compared with clinical assessment plus non-invasive tests (henceforth described as 'IUT' and 'no IUT' respectively) in women with SUI or stress-predominant mixed urinary incontinence (MUI) prior to surgery, and (ii) determining the expected net gain (ENG) from additional research.

METHODS

Study participants were women with SUI or stress-predominant MUI who had failed to respond to conservative treatments recruited from seven UK urogynaecology and female urology units. They were randomised to receive either 'IUT' or 'no IUT' before undergoing further treatment. Data from 218 women were used in the economic analysis. Cost utility, net benefit and value of information (VoI) analyses were performed within a randomised controlled pilot trial. Costs and quality-adjusted life years (QALYs) were estimated over 6 months to determine the incremental cost per QALY of 'IUT' compared to 'no IUT'. Net monetary benefit informed the VoI analysis. The VoI estimated the ENG and optimal sample size for a future definitive trial.

RESULTS

At 6 months, the mean difference in total average cost was £138 (p = 0.071) in favour of 'IUT'; there was no difference in QALYs estimated from the SF-12 (difference 0.004; p = 0.425) and EQ-5D-3L (difference - 0.004; p = 0.725); therefore, the probability of IUT being cost-effective remains uncertain. The estimated ENG was positive for further research to address this uncertainty with an optimal sample size of 404 women.

CONCLUSIONS

This is the largest economic evaluation of IUT. On average, up to 6 months after treatment, 'IUT' may be cost-saving compared to 'no IUT' because of the reduction in surgery following invasive investigation. However, uncertainty remains over the probability of 'IUT' being considered cost-effective, especially in the longer term. The VoI analysis indicated that further research would be of value.

TRIAL REGISTRATION

ISRCTN. ISRCTN71327395. Registered 7 June 2010.

Value of information methods to design a clinical trial in a small population to optimise a health economic utility function

Background

Most confirmatory randomised controlled clinical trials (RCTs) are designed with specified power, usually 80% or 90%, for a hypothesis test conducted at a given significance level, usually 2.5% for a one-sided test. Approval of the experimental treatment by regulatory agencies is then based on the result of such a significance test with other information to balance the risk of adverse events against the benefit of the treatment to future patients. In the setting of a rare disease, recruiting sufficient patients to achieve conventional error rates for clinically reasonable effect sizes may be infeasible, suggesting that the decision-making process should reflect the size of the target population.

Methods

We considered the use of a decision-theoretic value of information (VOI) method to obtain the optimal sample size and significance level for confirmatory RCTs in a range of settings. We assume the decision maker represents society. For simplicity we assume the primary endpoint to be normally distributed with unknown mean following some normal prior distribution representing information on the anticipated effectiveness of the therapy available before the trial. The method is illustrated by an application in an RCT in haemophilia A. We explicitly specify the utility in terms of improvement in primary outcome and compare this with the costs of treating patients, both financial and in terms of potential harm, during the trial and in the future.

Results

The optimal sample size for the clinical trial decreases as the size of the population decreases. For non-zero cost of treating future patients, either monetary or in terms of potential harmful effects, stronger evidence is required for approval as the population size increases, though this is not the case if the costs of treating future patients are ignored.

Conclusions

Decision-theoretic VOI methods offer a flexible approach with both type I error rate and power (or equivalently trial sample size) depending on the size of the future population for whom the treatment under investigation is intended. This might be particularly suitable for small populations when there is considerable information about the patient population.

Electronic supplementary material

The online version of this article (10.1186/s12874-018-0475-0) contains supplementary material, which is available to authorized users.

The Burden of the "False-Negatives" in Clinical Development: Analyses of Current and Alternative Scenarios and Corrective Measures

The “false‐negatives” of clinical development are the effective treatments wrongly determined ineffective. Statistical errors leading to “false‐negatives” are larger than those leading to “false‐positives,” especially in typically underpowered early‐phase trials. In addition, “false‐negatives” are usually eliminated from further testing, thereby limiting the information available on them. We simulated the impact of early‐phase power on economic productivity in three developmental scenarios. Scenario 1, representing the current status quo, assumed 50% statistical power at phase II and 90% at phase III. Scenario 2 assumed increased power (80%), and Scenario 3, increased stringency of alpha (1%) at phase II. Scenario 2 led, on average, to a 60.4% increase in productivity and 52.4% increase in profit. Scenario 3 had no meaningful advantages. Our results suggest that additional costs incurred by increasing the power of phase II studies are offset by the increase in productivity. We discuss the implications of our results and propose corrective measures.

Economic evaluation of factorial randomised controlled trials: challenges, methods and recommendations

Increasing numbers of economic evaluations are conducted alongside randomised controlled trials. Such studies include factorial trials, which randomise patients to different levels of two or more factors and can therefore evaluate the effect of multiple treatments alone and in combination. Factorial trials can provide increased statistical power or assess interactions between treatments, but raise additional challenges for trial‐based economic evaluations: interactions may occur more commonly for costs and quality‐adjusted life‐years (QALYs) than for clinical endpoints; economic endpoints raise challenges for transformation and regression analysis; and both factors must be considered simultaneously to assess which treatment combination represents best value for money. This article aims to examine issues associated with factorial trials that include assessment of costs and/or cost‐effectiveness, describe the methods that can be used to analyse such studies and make recommendations for health economists, statisticians and trialists. A hypothetical worked example is used to illustrate the challenges and demonstrate ways in which economic evaluations of factorial trials may be conducted, and how these methods affect the results and conclusions. Ignoring interactions introduces bias that could result in adopting a treatment that does not make best use of healthcare resources, while considering all interactions avoids bias but reduces statistical power. We also introduce the concept of the opportunity cost of ignoring interactions as a measure of the bias introduced by not taking account of all interactions. We conclude by offering recommendations for planning, analysing and reporting economic evaluations based on factorial trials, taking increased analysis costs into account. © 2017 The Authors. Statistics in Medicine published by John Wiley & Sons Ltd.

How Have Cancer Clinical Trial Eligibility Criteria Evolved Over Time?

Knowledge reuse of cancer trial designs may benefit from a temporal understanding of the evolution of the target populations of cancer studies over time. Therefore, we conducted a retrospective analysis of the trends of cancer trial eligibility criteria between 1999 and 2014. The yearly distributions of eligibility concepts for chemicals and drugs, procedures, observations, and medical conditions extracted from free-text eligibility criteria of 32,000 clinical trials for 89 cancer types were analyzed. We identified the concepts that trend upwards or downwards in all or selected cancer types, and the concepts that show anomalous trends for some cancers. Later, concept trends were studied in a disease-specific manner and illustrated for breast cancer. Criteria trends observed in this study are also validated and interpreted using evidence from the existing medical literature. This study contributes a method for concept trend analysis and original knowledge of the trends in cancer clinical trial eligibility criteria.

Pragmatic clinical trials: Emerging challenges and new roles for statisticians

Patients, clinicians, and policymakers alike need access to high-quality scientific evidence in order to make informed choices about health and healthcare, but the current national clinical trials enterprise is not yet optimally configured for the efficient creation and dissemination of such evidence. However, new technologies and methods hold significant potential for accelerating the rate at which we are able to translate raw findings gathered from both patient care and clinical research into actionable knowledge. We are now entering a period in which the quantitative sciences are emerging as the critical disciplines for advancing knowledge about health and healthcare, and statisticians will increasingly serve as critical mediators in transforming data into evidence. In this new, data-centric era, biostatisticians not only need to be expert at analyzing data but should also be involved directly in diverse efforts, including the review and analysis of research portfolios in order to optimize the relevance of research questions, the use of “quality by design” principles to improve reliability and validity of each individual trial, and the mining of aggregate knowledge derived from the clinical research enterprise as a whole. In order to meet these challenges, it is imperative that we (1) nurture and build the biostatistical workforce, (2) develop a deeper understanding of the biological and clinical context among statisticians, (3) facilitate collaboration among biostatisticians and other members of the clinical trials enterprise, (4) focus on communication skills in training and education programs, and (5) enhance the quantitative capacity of the research and clinical practice worlds.

Value of Information Analysis Informing Adoption and Research Decisions in a Portfolio of Health Care Interventions

Background: Value of information (VOI) analysis quantifies the value of additional research in reducing decision uncertainty. It addresses adoption and research decisions simultaneously by comparing the expected benefits and costs of research studies. Nevertheless, the application of this approach in practice remains limited. Objectives: To apply VOI analysis in health care interventions to guide adoption decisions, optimize trial design, and prioritize research. Methods: The analysis was from the perspective of Queensland Health, Australia. It included four interventions: clinically indicated catheter replacement, tissue adhesive for securing catheters, negative pressure wound therapy (NPWT) in caesarean sections, and nutritional support for preventing pressure ulcers. For each intervention, cost-effectiveness analysis was performed, decision uncertainty characterized, and VOI calculated using Monte Carlo simulations. The benefits and costs of additional research were considered together with the costs and consequences of acting now versus waiting for more information. All values are reported in 2014 Australian dollars (AU$). Results: All interventions were cost-effective, but with various levels of decision uncertainty. The current evidence is sufficient to support the adoption of clinically indicated catheter replacement. For the tissue adhesive, an additional study before adoption is worthwhile with a four-arm trial of 220 patients per arm. Additional research on NPWT before adoption is worthwhile with a two-arm trial of 200 patients per arm. Nutritional support should be adopted with a two-arm trial of 1200 patients per arm. Based on the expected net monetary benefits, the studies were ranked as follows: 1) NPWT (AU$1.2 million), 2) tissue adhesive (AU$0.3 milliion), and 3) nutritional support (AU$0.1 million). Conclusions: VOI analysis is a useful and practical approach to inform adoption and research decisions. Efforts should be focused on facilitating its integration into decision making frameworks.

Clinical decision making and the expected value of information

Background The results of the HOPE study, a randomized clinical trial, provide strong evidence that 1) ramipril prevents the composite outcome of cardiovascular death, myocardial infarction or stroke in patients who are at high risk of a cardiovascular event and 2) ramipril is cost-effective at a threshold willingness-to-pay of $10 000 to prevent an event of the composite outcome. In this report the concept of the expected value of information is used to determine if the information provided by the HOPE study is sufficient for decision making in the US and Canada.

Methods and results Using the cost-effectiveness data from a clinical trial, or from a meta-analysis of several trials, one can determine, based on the number of future patients that would benefit from the health technology under investigation, the expected value of sample information (EVSI) of a future trial as a function of proposed sample size. If the EVSI exceeds the cost for any particular sample size then the current information is insufficient for decision making and a future trial is indicated. If, on the other hand, there is no sample size for which the EVSI exceeds the cost, then there is sufficient information for decision making and no future trial is required. Using the data from the HOPE study these concepts are applied for various assumptions regarding the fixed and variable cost of a future trial and the number of patients who would benefit from ramipril.

Conclusions Expected value of information methods provide a decision-analytic alternative to the standard likelihood methods for assessing the evidence provided by cost-effectiveness data from randomized clinical trials. Clinical Trials 2007; 4: 279—285. http://ctj.sagepub.com

A feasibility study incorporating a pilot randomised controlled trial of oral feeding plus pre-treatment gastrostomy tube versus oral feeding plus as-needed nasogastric tube feeding in patients undergoing chemoradiation for head and neck cancer (TUBE trial): study protocol

BACKGROUND

There are 7000 new cases of head and neck squamous cell cancers (HNSCC) treated by the NHS each year. Stage III and IV HNSCC can be treated non-surgically by radio therapy (RT) or chemoradiation therapy (CRT). CRT can affect eating and drinking through a range of side effects with 90 % of patients undergoing this treatment requiring nutritional support via gastrostomy (G) or nasogastric (NG) tube feeding. Long-term dysphagia following CRT is a primary concern for patients. The effect of enteral feeding routes on swallowing function is not well understood, and the two feeding methods have, to date, not been compared to assess which leads to a better patient outcome. The purpose of this study is to explore the feasibility of conducting a randomised controlled trial (RCT) comparing these two options with particular emphasis on patient willingness to be randomised and clinician willingness to approach eligible patients.

METHODS/DESIGN

This is a mixed methods multicentre study to establish the feasibility of a randomised controlled trial comparing oral feeding plus pre-treatment gastrostomy versus oral feeding plus as required nasogastric tube feeding in patients with HNSCC. A total of 60 participants will be randomised to the two arms of the study (1:1 ratio). The primary outcome of feasibility is a composite of recruitment (willingness to randomise and be randomised) and retention. A qualitative process evaluation investigating patient, family and friends and staff experiences of trial participation will also be conducted alongside an economic modelling exercise to synthesise available evidence and provide estimates of cost-effectiveness and value of information. Participants will be assessed at baseline (pre-randomisation), during CRT weekly, 3 months and 6 months.

DISCUSSION

Clinicians are in equipoise over the enteral feeding options for patients being treated with CRT. Swallowing outcomes have been identified as a top priority for patients following treatment and this trial would inform a future larger scale RCT in this area to inform best practice.

TRIAL REGISTRATION

ISRCTN48569216.

Efficient Value of Information Calculation Using a Nonparametric Regression Approach: An Applied Perspective

BACKGROUND

Value-of-information (VOI) analysis provides an analytical framework to assess whether obtaining additional evidence is worthwhile to reduce decision uncertainty. The reporting of VOI measures, particularly the expected value of perfect parameter information (EVPPI) and the expected value of sample information (EVSI), is limited because of the computational burden associated with typical two-level Monte-Carlo-based solution. Recently, a nonparametric regression approach was proposed that allows the estimation of multiparameter EVPPI and EVSI directly from a probabilistic sensitivity analysis sample.

OBJECTIVES

To demonstrate the value of the nonparametric regression approach in calculating VOI measures in real-world cases and to compare its performance with the standard approach of the Monte-Carlo simulation.

METHODS

We used the regression approach to calculate EVPPI and EVSI in two models, and compared the results with the estimates obtained via the standard Monte-Carlo simulation.

RESULTS

The VOI values from the two approaches were very close; computation using the regression method, however, was faster.

CONCLUSION

The nonparametric regression approach provides an efficient and easy-to-implement alternative for EVPPI and EVSI calculation in economic models.

Non-invasive versus invasive management in patients with prior coronary artery bypass surgery with a non-ST segment elevation acute coronary syndrome: study design of the pilot randomised controlled trial and registry (CABG-ACS)

Introduction

There is an evidence gap about how to best treat patients with prior coronary artery bypass grafts (CABGs) presenting with non-ST segment elevation acute coronary syndromes (NSTE-ACS) because historically, these patients were excluded from pivotal randomised trials. We aim to undertake a pilot trial of routine non-invasive management versus routine invasive management in patients with NSTE-ACS with prior CABG and optimal medical therapy during routine clinical care. Our trial is a proof-of-concept study for feasibility, safety, potential efficacy and health economic modelling. We hypothesise that a routine invasive approach in patients with NSTE-ACS with prior CABG is not superior to a non-invasive approach with optimal medical therapy.

Methods and analysis

60 patients will be enrolled in a randomised clinical trial in 4 hospitals. A screening log will be prospectively completed. Patients not randomised due to lack of eligibility criteria and/or patient or physician preference and who give consent will be included in a registry. We will gather information about screening, enrolment, eligibility, randomisation, patient characteristics and adverse events (including post-discharge). The primary efficacy outcome is the composite of all-cause mortality, rehospitalisation for refractory ischaemia/angina, myocardial infarction and hospitalisation for heart failure. The primary safety outcome is the composite of bleeding, stroke, procedure-related myocardial infarction and worsening renal function. Health status will be assessed using EuroQol 5 Dimensions (EQ-5D) assessed at baseline and 6 monthly intervals, for at least 18 months.

Trial registration number

NCT01895751 (ClinicalTrials.gov).

Development and Evaluation of an Approach to Using Value of Information Analyses for Real-Time Prioritization Decisions Within SWOG, a Large Cancer Clinical Trials Cooperative Group

OBJECTIVE

Value of information (VOI) analyses can align research with areas with the greatest potential impact on patient outcome, but questions remain concerning the feasibility and acceptability of these approaches to inform prioritization decisions. Our objective was to develop a process for calculating VOI in "real time" to inform trial funding decisions within SWOG, a large cancer clinical trials group.

METHODS

We developed an efficient and scalable VOI modeling approach using a selected sample of 9 randomized phase II/III trial proposals from the Breast, Gastrointestinal, and Genitourinary Disease Committees reviewed by SWOG's leadership between 2008 and 2013. There was bidirectional communication between SWOG investigators and the research team throughout the modeling development. Partial expected value of sample information for the treatment effect evaluated by the proposed trial's primary endpoint was calculated using Monte Carlo simulation.

RESULTS

We derived prior uncertainty in the treatment effect estimate from the sample size calculations. Our process was feasible for 8 of 9 trial proposals and efficient: the time required of 1 researcher was <1 week per proposal. We accommodated stakeholder input primarily by deconstructing VOI metrics into expected health benefits and incremental healthcare costs and assuming treatment decisions within our simulations were based on health benefits. Following customization, feedback from over 200 SWOG members was positive regarding the overall VOI framework, specific retrospective results, and potential for VOI analyses to inform future trial proposal evaluations.

CONCLUSIONS

We developed an efficient and customized process to calculate the expected VOI of cancer clinical trials that is feasible for use in decision making and acceptable to investigators. Prospective use and evaluation of this approach is currently underway within SWOG.

When Quality Beats Quantity: Decision Theory, Drug Discovery, and the Reproducibility Crisis

A striking contrast runs through the last 60 years of biopharmaceutical discovery, research, and development. Huge scientific and technological gains should have increased the quality of academic science and raised industrial R&D efficiency. However, academia faces a "reproducibility crisis"; inflation-adjusted industrial R&D costs per novel drug increased nearly 100 fold between 1950 and 2010; and drugs are more likely to fail in clinical development today than in the 1970s. The contrast is explicable only if powerful headwinds reversed the gains and/or if many "gains" have proved illusory. However, discussions of reproducibility and R&D productivity rarely address this point explicitly. The main objectives of the primary research in this paper are: (a) to provide quantitatively and historically plausible explanations of the contrast; and (b) identify factors to which R&D efficiency is sensitive. We present a quantitative decision-theoretic model of the R&D process. The model represents therapeutic candidates (e.g., putative drug targets, molecules in a screening library, etc.) within a "measurement space", with candidates' positions determined by their performance on a variety of assays (e.g., binding affinity, toxicity, in vivo efficacy, etc.) whose results correlate to a greater or lesser degree. We apply decision rules to segment the space, and assess the probability of correct R&D decisions. We find that when searching for rare positives (e.g., candidates that will successfully complete clinical development), changes in the predictive validity of screening and disease models that many people working in drug discovery would regard as small and/or unknowable (i.e., an 0.1 absolute change in correlation coefficient between model output and clinical outcomes in man) can offset large (e.g., 10 fold, even 100 fold) changes in models' brute-force efficiency. We also show how validity and reproducibility correlate across a population of simulated screening and disease models. We hypothesize that screening and disease models with high predictive validity are more likely to yield good answers and good treatments, so tend to render themselves and their diseases academically and commercially redundant. Perhaps there has also been too much enthusiasm for reductionist molecular models which have insufficient predictive validity. Thus we hypothesize that the average predictive validity of the stock of academically and industrially "interesting" screening and disease models has declined over time, with even small falls able to offset large gains in scientific knowledge and brute-force efficiency. The rate of creation of valid screening and disease models may be the major constraint on R&D productivity.

Valuing Trial Designs from a Pharmaceutical Perspective Using Value-Based Pricing

Our aim was to adapt the traditional framework for expected net benefit of sampling (ENBS) to be more compatible with drug development trials from the pharmaceutical perspective. We modify the traditional framework for conducting ENBS and assume that the price of the drug is conditional on the trial outcomes. We use a value-based pricing (VBP) criterion to determine price conditional on trial data using Bayesian updating of cost-effectiveness (CE) model parameters. We assume that there is a threshold price below which the company would not market the new intervention. We present a case study in which a phase III trial sample size and trial duration are varied. For each trial design, we sampled 10,000 trial outcomes and estimated VBP using a CE model. The expected commercial net benefit is calculated as the expected profits minus the trial costs. A clinical trial with shorter follow-up, and larger sample size, generated the greatest expected commercial net benefit. Increasing the duration of follow-up had a modest impact on profit forecasts. Expected net benefit of sampling can be adapted to value clinical trials in the pharmaceutical industry to optimise the expected commercial net benefit. However, the analyses can be very time consuming for complex CE models.