Expected Value of Sample Information Calculations in Medical Decision Modeling

Value of Information for Clinical Trial Design: The Importance of Considering All Relevant Comparators

Value of Information (VOI) analyses calculate the economic value that could be generated by obtaining further information to reduce uncertainty in a health economic decision model. VOI has been suggested as a tool for research prioritisation and trial design as it can highlight economically valuable avenues for future research. Recent methodological advances have made it increasingly feasible to use VOI in practice for research; however, there are critical differences between the VOI approach and the standard methods used to design research studies such as clinical trials. We aimed to highlight key differences between the research design approach based on VOI and standard clinical trial design methods, in particular the importance of considering the full decision context. We present two hypothetical examples to demonstrate that VOI methods are only accurate when (1) all feasible comparators are included in the decision model when designing research, and (2) all comparators are retained in the decision model once the data have been collected and a final treatment recommendation is made. Omitting comparators from either the design or analysis phase of research when using VOI methods can lead to incorrect trial designs and/or treatment recommendations. Overall, we conclude that incorrectly specifying the health economic model by ignoring potential comparators can lead to misleading VOI results and potentially waste scarce research resources.

Estimating the value of future research into thromboprophylaxis for women during pregnancy and after delivery: a value of information analysis

BACKGROUND

Risk assessment models (RAMs) are used to select women at increased risk of venous thromboembolism (VTE) during pregnancy and the puerperium for thromboprophylaxis.

OBJECTIVES

To estimate the value of potential future studies that would reduce the decision uncertainty associated with offering thromboprophylaxis according to available RAMs in the following groups: high-risk antepartum women (eg, prior VTE), unselected postpartum women, and postpartum women with risk factors (obesity or cesarean delivery).

METHODS

A decision-analytic model was developed to simulate clinical outcomes, lifetime costs, and quality-adjusted life-years for different thromboprophylaxis strategies, including thromboprophylaxis for all, thromboprophylaxis for none, and RAM-based thromboprophylaxis. The expected value of perfect information analysis was used to determine which factors are associated with high decision uncertainty. The value of future research studies was estimated using expected value of sample information analysis. Costs were assessed from a health and social services perspective.

RESULTS

The expected value of perfect information analysis identified high decision uncertainty for high-risk antepartum women (£21.8 million) and obese postpartum women (£13.4 million), which was largely attributable to uncertainty regarding the effectiveness of thromboprophylaxis in reducing VTE. A randomized controlled trial of thromboprophylaxis compared with none in obese postpartum women is likely to have substantial value (£2.8 million; 300 participants per arm). A trial in women with previous VTE would have higher value but would be less acceptable.

CONCLUSION

Future research should focus on estimating the effectiveness of thromboprophylaxis in obese postpartum women with additional risk factors who have not had a previous VTE.

Thromboprophylaxis during pregnancy and the puerperium: a systematic review and economic evaluation to estimate the value of future research

Background

Pharmacological prophylaxis to prevent venous thromboembolism is currently recommended for women assessed as being at high risk of venous thromboembolism during pregnancy or in the 6 weeks after delivery (the puerperium). The decision to provide thromboprophylaxis involves weighing the benefits, harms and costs, which vary according to the individual's venous thromboembolism risk. It is unclear whether the United Kingdom's current risk stratification approach could be improved by further research.

Objectives

To quantify the current decision uncertainty associated with selecting women who are pregnant or in the puerperium for thromboprophylaxis and to estimate the value of one or more potential future studies that would reduce that uncertainty, while being feasible and acceptable to patients and clinicians.

Methods

A decision-analytic model was developed which was informed by a systematic review of risk assessment models to predict venous thromboembolism in women who are pregnant or in the puerperium. Expected value of perfect information analysis was used to determine which factors are associated with high decision uncertainty and should be the target of future research. To find out whether future studies would be acceptable and feasible, we held workshops with women who have experienced a blood clot or have been offered blood-thinning drugs and surveyed healthcare professionals. Expected value of sample information analysis was used to estimate the value of potential future research studies.

Results

The systematic review included 17 studies, comprising 19 unique externally validated risk assessment models and 1 internally validated model. Estimates of sensitivity and specificity were highly variable ranging from 0% to 100% and 5% to 100%, respectively. Most studies had unclear or high risk of bias and applicability concerns. The decision analysis found that there is substantial decision uncertainty regarding the use of risk assessment models to select high-risk women for antepartum prophylaxis and obese postpartum women for postpartum prophylaxis. The main source of decision uncertainty was uncertainty around the effectiveness of thromboprophylaxis for preventing venous thromboembolism in women who are pregnant or in the puerperium. We found that a randomised controlled trial of thromboprophylaxis in obese postpartum women is likely to have substantial value and is more likely to be acceptable and feasible than a trial recruiting women who have had a previous venous thromboembolism. In unselected postpartum women and women following caesarean section, the poor performance of risk assessment models meant that offering prophylaxis based on these models had less favourable cost effectiveness with lower decision uncertainty.

Limitations

The performance of the risk assessment model for obese postpartum women has not been externally validated.

Conclusions

Future research should focus on estimating the efficacy of pharmacological thromboprophylaxis in pregnancy and the puerperium, and clinical trials would be more acceptable in women who have not had a previous venous thromboembolism.

Study registration

This study is registered as PROSPERO CRD42020221094.

Funding

This award was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme (NIHR award ref: NIHR131021) and is published in full in Health Technology Assessment; Vol. 28, No. 9. See the NIHR Funding and Awards website for further award information.

Consolidated Health Economic Evaluation Reporting Standards - Value of Information (CHEERS-VOI): Explanation and Elaboration

OBJECTIVES

Although the ISPOR Value of Information (VOI) Task Force's reports outline VOI concepts and provide good-practice recommendations, there is no guidance for reporting VOI analyses. VOI analyses are usually performed alongside economic evaluations for which the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) 2022 Statement provides reporting guidelines. Thus, we developed the CHEERS-VOI checklist to provide reporting guidance and checklist to support the transparent, reproducible, and high-quality reporting of VOI analyses.

METHODS

A comprehensive literature review generated a list of 26 candidate reporting items. These candidate items underwent a Delphi procedure with Delphi participants through 3 survey rounds. Participants rated each item on a 9-point Likert scale to indicate its relevance when reporting the minimal, essential information about VOI methods and provided comments. The Delphi results were reviewed at 2-day consensus meetings and the checklist was finalized using anonymous voting.

RESULTS

We had 30, 25, and 24 Delphi respondents in rounds 1, 2, and 3, respectively. After incorporating revisions recommended by the Delphi participants, all 26 candidate items proceeded to the 2-day consensus meetings. The final CHEERS-VOI checklist includes all CHEERS items, but 7 items require elaboration when reporting VOI. Further, 6 new items were added to report information relevant only to VOI (eg, VOI methods applied).

CONCLUSIONS

The CHEERS-VOI checklist should be used when a VOI analysis is performed alongside economic evaluations. The CHEERS-VOI checklist will help decision makers, analysts and peer reviewers in the assessment and interpretation of VOI analyses and thereby increase transparency and rigor in decision making.

General-Purpose Methods for Simulating Survival Data for Expected Value of Sample Information Calculations

BACKGROUND

Expected value of sample information (EVSI) quantifies the expected value to a decision maker of reducing uncertainty by collecting additional data. EVSI calculations require simulating plausible data sets, typically achieved by evaluating quantile functions at random uniform numbers using standard inverse transform sampling (ITS). This is straightforward when closed-form expressions for the quantile function are available, such as for standard parametric survival models, but these are often unavailable when assuming treatment effect waning and for flexible survival models. In these circumstances, the standard ITS method could be implemented by numerically evaluating the quantile functions at each iteration in a probabilistic analysis, but this greatly increases the computational burden. Thus, our study aims to develop general-purpose methods that standardize and reduce the computational burden of the EVSI data-simulation step for survival data.

METHODS

We developed a discrete sampling method and an interpolated ITS method for simulating survival data from a probabilistic sample of survival probabilities over discrete time units. We compared the general-purpose and standard ITS methods using an illustrative partitioned survival model with and without adjustment for treatment effect waning.

RESULTS

The discrete sampling and interpolated ITS methods agree closely with the standard ITS method, with the added benefit of a greatly reduced computational cost in the scenario with adjustment for treatment effect waning.

CONCLUSIONS

We present general-purpose methods for simulating survival data from a probabilistic sample of survival probabilities that greatly reduce the computational burden of the EVSI data-simulation step when we assume treatment effect waning or use flexible survival models. The implementation of our data-simulation methods is identical across all possible survival models and can easily be automated from standard probabilistic decision analyses.

HIGHLIGHTS

Expected value of sample information (EVSI) quantifies the expected value to a decision maker of reducing uncertainty through a given data collection exercise, such as a randomized clinical trial. In this article, we address the problem of computing EVSI when we assume treatment effect waning or use flexible survival models, by developing general-purpose methods that standardize and reduce the computational burden of the EVSI data-generation step for survival data.We developed 2 methods for simulating survival data from a probabilistic sample of survival probabilities over discrete time units, a discrete sampling method and an interpolated inverse transform sampling method, which can be combined with a recently proposed nonparametric EVSI method to accurately estimate EVSI for collecting survival data.Our general-purpose data-simulation methods greatly reduce the computational burden of the EVSI data-simulation step when we assume treatment effect waning or use flexible survival models. The implementation of our data-simulation methods is identical across all possible survival models and can therefore easily be automated from standard probabilistic decision analyses.

Constructing Relative Effect Priors for Research Prioritization and Trial Design: A Meta-epidemiological Analysis

BACKGROUND

Bayesian methods have potential for efficient design of randomized clinical trials (RCTs) by incorporating existing evidence. Furthermore, value of information (VOI) methods estimate the value of reducing decision uncertainty, aiding transparent research prioritization. These methods require a prior distribution describing current uncertainty in key parameters, such as relative treatment effect (RTE). However, at the time of designing and commissioning research, there may be no data to base the prior on. The aim of this article is to present methods to construct priors for RTEs based on a collection of previous RCTs.

METHODS

We developed 2 Bayesian hierarchical models that captured variability in RTE between studies within disease area accounting for study characteristics. We illustrate the methods using a data set of 743 published RCTs across 9 disease areas to obtain predictive distributions for RTEs for a range of disease areas. We illustrate how the priors from such an analysis can be used in a VOI analysis for an RCT in bladder cancer and compare the results with those using an uninformative prior.

RESULTS

For most disease areas, the predicted RTE favored new interventions over comparators. The predicted effects and uncertainty differed across the 9 disease areas. VOI analysis showed that the expected value of research is much lower with our empirically derived prior compared with an uninformative prior.

CONCLUSIONS

This study demonstrates a novel approach to generating informative priors that can be used to aid research prioritization and trial design. The methods can also be used to combine RCT evidence with expert opinion. Further work is needed to create a rich database of RCT evidence that can be used to form off-the-shelf priors.

HIGHLIGHTS

Bayesian methods have potential to aid the efficient design of randomized clinical trials (RCTs) by incorporating existing evidence. Value-of-information (VOI) methods can be used to aid research prioritization by calculating the value of current decision uncertainty.These methods require a distribution describing current uncertainty in key parameters, that is, "prior distributions."This article demonstrates a methodology to estimate prior distributions for relative treatment effects (odds and hazard ratios) estimated from a collection of previous RCTs.These results may be combined with expert elicitation to facilitate 1) value-of-information methods to prioritize research or 2) Bayesian methods for research design.

Use of Bayesian decision analysis to maximize value in patient-centered randomized clinical trials in Parkinson's disease

A fixed one-sided significance level of 5% is commonly used to interpret the statistical significance of randomized clinical trial (RCT) outcomes. While it is necessary to reduce the false positive rate, the threshold used could be chosen quantitatively and transparently to specifically reflect patient preferences regarding benefit-risk tradeoffs as well as other considerations. How can patient preferences be explicitly incorporated into RCTs in Parkinson's disease (PD), and what is the impact on statistical thresholds for device approval? In this analysis, we apply Bayesian decision analysis (BDA) to PD patient preference scores elicited from survey data. BDA allows us to choose a sample size (n ) and significance level (α ) that maximizes the overall expected value to patients of a balanced two-arm fixed-sample RCT, where the expected value is computed under both null and alternative hypotheses. For PD patients who had previously received deep brain stimulation (DBS) treatment, the BDA-optimal significance levels fell between 4.0% and 10.0%, similar to or greater than the traditional value of 5%. Conversely, for patients who had never received DBS, the optimal significance level ranged from 0.2% to 4.4%. In both of these populations, the optimal significance level increased with the severity of the patients' cognitive and motor function symptoms. By explicitly incorporating patient preferences into clinical trial designs and the regulatory decision-making process, BDA provides a quantitative and transparent approach to combine clinical and statistical significance. For PD patients who have never received DBS treatment, a 5% significance threshold may not be conservative enough to reflect their risk-aversion level. However, this study shows that patients who previously received DBS treatment present a higher tolerance to accept therapeutic risks in exchange for improved efficacy which is reflected in a higher statistical threshold.

A Tutorial on Time-Dependent Cohort State-Transition Models in R Using a Cost-Effectiveness Analysis Example

In an introductory tutorial, we illustrated building cohort state-transition models (cSTMs) in R, where the state transition probabilities were constant over time. However, in practice, many cSTMs require transitions, rewards, or both to vary over time (time dependent). This tutorial illustrates adding 2 types of time dependence using a previously published cost-effectiveness analysis of multiple strategies as an example. The first is simulation-time dependence, which allows for the transition probabilities to vary as a function of time as measured since the start of the simulation (e.g., varying probability of death as the cohort ages). The second is state-residence time dependence, allowing for history by tracking the time spent in any particular health state using tunnel states. We use these time-dependent cSTMs to conduct cost-effectiveness and probabilistic sensitivity analyses. We also obtain various epidemiological outcomes of interest from the outputs generated from the cSTM, such as survival probability and disease prevalence, often used for model calibration and validation. We present the mathematical notation first, followed by the R code to execute the calculations. The full R code is provided in a public code repository for broader implementation.

A Unified Approach for Outliers and Influential Data Detection - The Value of Information in Retrospect

Identifying influential and outlying data is important as it would guide the effective collection of future data and the proper use of existing information. We develop a unified approach for outlier detection and influence analysis. Our proposed method is grounded in the intuitive value of information concepts and has a distinct advantage in interpretability and flexibility when compared to existing methods: it decomposes the data influence into the leverage effect (expected to be influential) and the outlying effect (surprisingly more influential than being expected); and it applies to all decision problems such as estimation, prediction, and hypothesis testing. We study the theoretical properties of three value of information quantities, establish the relationship between the proposed measures and classic measures in the linear regression setting, and provide real data analysis examples of how to apply the new value of information approach in the cases of linear regression, generalized linear mixed model, and hypothesis testing.

A Closed-Form EVSI Expression for a Multinomial Data-Generating Process

This paper derives analytic expressions for the expected value of sample information (EVSI), the expected value of distribution information, and the optimal sample size when data consists of independent draws from a bounded sequence of integers. Because of the challenges of creating tractable EVSI expressions, most existing work valuing data does so in one of three ways: (1) analytically through closed-form expressions on the upper bound of the value of data, (2) calculating the expected value of data using numerical comparisons of decisions made using simulated data to optimal decisions for which the underlying data distribution is known, or (3) using variance reduction as proxy for the uncertainty reduction that accompanies more data. For the very flexible case of modeling integer-valued observations using a multinomial data-generating process with Dirichlet prior, this paper develops expressions that (1) generalize existing beta-binomial computations, (2) do not require prior knowledge of some underlying “true” distribution, and (3) can be computed prior to the collection of any sample data.

Estimated Monetary Value of Future Research Clarifying Uncertainties Around the Optimal Adult Hearing Screening Schedule

Importance

Adult hearing screening is not routinely performed, and most individuals with hearing loss (HL) have never had their hearing tested as adults.

Objective

To project the monetary value of future research clarifying uncertainties around the optimal adult hearing screening schedule.

Design, Setting, and Participants

In this economic evaluation, a validated decision model of HL (DeciBHAL-US: Decision model of the Burden of Hearing loss Across the Lifespan) was used to simulate current detection and treatment of HL vs hearing screening schedules. Key model inputs included HL incidence (0.06%-10.42%/y), hearing aid uptake (0.54%-8.14%/y), screening effectiveness (1.62 × hearing aid uptake), utility benefits of hearing aids (+0.11), and hearing aid device costs ($3690). Distributions to model parameters for probabilistic uncertainty analysis were assigned. The expected value of perfect information (EVPI) and expected value of partial perfect information (EVPPI) using a willingness to pay of $100 000 per quality-adjusted life-year (QALY) was estimated. The EVPI and EVPPI estimate the upper bound of the dollar value of future research. This study was based on 40-year-old persons over their remaining lifetimes in a US primary care setting.

Exposures

Screening schedules beginning at ages 45, 55, 65, and 75 years, and frequencies of every 1 or 5 years.

Main Outcomes and Measures

The main outcomes were QALYs and costs (2020 US dollars) from a health system perspective.

Results

The average incremental cost-effectiveness ratio for yearly screening beginning at ages 55 to 75 years ranged from $39 200 to $80 200/QALY. Yearly screening beginning at age 55 years was the optimal screening schedule in 38% of probabilistic uncertainty analysis simulations. The population EVPI, or value of reducing all uncertainty, was $8.2 to $12.6 billion varying with willingness to pay and the EVPPI, or value of reducing all screening effectiveness uncertainty, was $2.4 billion.

Conclusions and Relevance

In this economic evaluation of US adult hearing screening, large uncertainty around the optimal adult hearing screening schedule was identified. Future research on hearing screening has a high potential value so is likely justified.

Where to Invest Project Efforts for Greater Benefit: A Framework for Management Performance Mapping with Examples for Potato Seed Health

Policymakers and donors often need to identify the locations where technologies are most likely to have important effects, to increase the benefits from agricultural development or extension efforts. Higher-quality information may help to target the high-benefit locations, but often actions are needed with limited information. The value of information (VOI) in this context is formalized by evaluating the results of decision making guided by a set of specific information compared with the results of acting without considering that information. We present a framework for management performance mapping that includes evaluating the VOI for decision making about geographic priorities in regional intervention strategies, in case studies of Andean and Kenyan potato seed systems. We illustrate the use of recursive partitioning, XGBoost, and Bayesian network models to characterize the relationships among seed health and yield responses and environmental and management predictors used in studies of seed degeneration. These analyses address the expected performance of an intervention based on geographic predictor variables. In the Andean example, positive selection of seed from asymptomatic plants was more effective at high altitudes in Ecuador. In the Kenyan example, there was the potential to target locations with higher technology adoption rates and with higher potato cropland connectivity, i.e., a likely more important role in regional epidemics. Targeting training to high management performance areas would often provide more benefits than would random selection of target areas. We illustrate how assessing the VOI can contribute to targeted development programs and support a culture of continuous improvement for interventions.[Formula: see text] Copyright © 2022 The Author(s). This is an open access article distributed under the CC BY 4.0 International license.

A Conceptual Framework for Life-Cycle Health Technology Assessment

OBJECTIVES

Health technology assessment (HTA) uses evidence appraisal and synthesis with economic evaluation to inform adoption decisions. Standard HTA processes sometimes struggle to (1) support decisions that involve significant uncertainty and (2) encourage continued generation of and adaptation to new evidence. We propose the life-cycle (LC)-HTA framework, addressing these challenges by providing additional tools to decision makers and improving outcomes for all stakeholders.

METHODS

Under the LC-HTA framework, HTA processes align to LC management. LC-HTA introduces changes in HTA methods to minimize analytic time while optimizing decision certainty. Where decision uncertainty exists, we recommend risk-based pricing and research-oriented managed access (ROMA). Contractual procurement agreements define the terms of reassessment and provide additional decision options to HTA agencies. LC-HTA extends value-of-information methods to inform ROMA agreements, leveraging routine, administrative data, and registries to reduce uncertainty.

RESULTS

LC-HTA enables the adoption of high-value high-risk innovations while improving health system sustainability through risk-sharing and reducing uncertainty. Responsiveness to evolving evidence is improved through contractually embedded decision rules to simplify reassessment. ROMA allows conditional adoption to obtain additional information, with confidence that the net value of that adoption decision is positive.

CONCLUSIONS

The LC-HTA framework improves outcomes for patients, sponsors, and payers. Patients benefit through earlier access to new technologies. Payers increase the value of the technologies they invest in and gain mechanisms to review investments. Sponsors benefit through greater certainty in outcomes related to their investment, swifter access to markets, and greater opportunities to demonstrate value.

An Efficient Method for Computing Expected Value of Sample Information for Survival Data from an Ongoing Trial

BACKGROUND

Decisions about new health technologies are increasingly being made while trials are still in an early stage, which may result in substantial uncertainty around key decision drivers such as estimates of life expectancy and time to disease progression. Additional data collection can reduce uncertainty, and its value can be quantified by computing the expected value of sample information (EVSI), which has typically been described in the context of designing a future trial. In this article, we develop new methods for computing the EVSI of extending an existing trial's follow-up, first for an assumed survival model and then extending to capture uncertainty about the true survival model.

METHODS

We developed a nested Markov Chain Monte Carlo procedure and a nonparametric regression-based method. We compared the methods by computing single-model and model-averaged EVSI for collecting additional follow-up data in 2 synthetic case studies.

RESULTS

There was good agreement between the 2 methods. The regression-based method was fast and straightforward to implement, and scales easily to include any number of candidate survival models in the model uncertainty case. The nested Monte Carlo procedure, on the other hand, was extremely computationally demanding when we included model uncertainty.

CONCLUSIONS

We present a straightforward regression-based method for computing the EVSI of extending an existing trial's follow-up, both where a single known survival model is assumed and where we are uncertain about the true survival model. EVSI for ongoing trials can help decision makers determine whether early patient access to a new technology can be justified on the basis of the current evidence or whether more mature evidence is needed.

HIGHLIGHTS

Decisions about new health technologies are increasingly being made while trials are still in an early stage, which may result in substantial uncertainty around key decision drivers such as estimates of life-expectancy and time to disease progression. Additional data collection can reduce uncertainty, and its value can be quantified by computing the expected value of sample information (EVSI), which has typically been described in the context of designing a future trial.In this article, we have developed new methods for computing the EVSI of extending a trial's follow-up, both where a single known survival model is assumed and where we are uncertain about the true survival model. We extend a previously described nonparametric regression-based method for computing EVSI, which we demonstrate in synthetic case studies is fast, straightforward to implement, and scales easily to include any number of candidate survival models in the EVSI calculations.The EVSI methods that we present in this article can quantify the need for collecting additional follow-up data before making an adoption decision given any decision-making context.

Value of Information Analysis in Models to Inform Health Policy

Value of information (VoI) is a decision-theoretic approach to estimating the expected benefits from collecting further information of different kinds, in scientific problems based on combining one or more sources of data. VoI methods can assess the sensitivity of models to different sources of uncertainty and help to set priorities for further data collection. They have been widely applied in healthcare policy making, but the ideas are general to a range of evidence synthesis and decision problems. This article gives a broad overview of VoI methods, explaining the principles behind them, the range of problems that can be tackled with them, and how they can be implemented, and discusses the ongoing challenges in the area.

Uncertainty and the Value of Information in Risk Prediction Modeling

Background

Because of the finite size of the development sample, predicted probabilities from a risk prediction model are inevitably uncertain. We apply value-of-information methodology to evaluate the decision-theoretic implications of prediction uncertainty.

Methods

Adopting a Bayesian perspective, we extend the definition of the expected value of perfect information (EVPI) from decision analysis to net benefit calculations in risk prediction. In the context of model development, EVPI is the expected gain in net benefit by using the correct predictions as opposed to predictions from a proposed model. We suggest bootstrap methods for sampling from the posterior distribution of predictions for EVPI calculation using Monte Carlo simulations. We used subsets of data of various sizes from a clinical trial for predicting mortality after myocardial infarction to show how EVPI changes with sample size.

Results

With a sample size of 1000 and at the prespecified threshold of 2% on predicted risks, the gains in net benefit using the proposed and the correct models were 0.0006 and 0.0011, respectively, resulting in an EVPI of 0.0005 and a relative EVPI of 87%. EVPI was zero only at unrealistically high thresholds (>85%). As expected, EVPI declined with larger samples. We summarize an algorithm for incorporating EVPI calculations into the commonly used bootstrap method for optimism correction.

Conclusion

The development EVPI can be used to decide whether a model can advance to validation, whether it should be abandoned, or whether a larger development sample is needed. Value-of-information methods can be applied to explore decision-theoretic consequences of uncertainty in risk prediction and can complement inferential methods in predictive analytics. R code for implementing this method is provided.

Simulating Study Data to Support Expected Value of Sample Information Calculations: A Tutorial

The expected value of sample information (EVSI) can be used to prioritize avenues for future research and design studies that support medical decision making and offer value for money spent. EVSI is calculated based on 3 key elements. Two of these, a probabilistic model-based economic evaluation and updating model uncertainty based on simulated data, have been frequently discussed in the literature. By contrast, the third element, simulating data from the proposed studies, has received little attention. This tutorial contributes to bridging this gap by providing a step-by-step guide to simulating study data for EVSI calculations. We discuss a general-purpose algorithm for simulating data and demonstrate its use to simulate 3 different outcome types. We then discuss how to induce correlations in the generated data, how to adjust for common issues in study implementation such as missingness and censoring, and how individual patient data from previous studies can be leveraged to undertake EVSI calculations. For all examples, we provide comprehensive code written in the R language and, where possible, Excel spreadsheets in the supplementary materials. This tutorial facilitates practical EVSI calculations and allows EVSI to be used to prioritize research and design studies.

Calculating Expected Value of Sample Information Adjusting for Imperfect Implementation

BACKGROUND

The expected value of sample information (EVSI) calculates the value of collecting additional information through a research study with a given design. However, standard EVSI analyses do not account for the slow and often incomplete implementation of the treatment recommendations that follow research. Thus, standard EVSI analyses do not correctly capture the value of the study. Previous research has developed measures to calculate the research value while adjusting for implementation challenges, but estimating these measures is a challenge.

METHODS

Based on a method that assumes the implementation level is related to the strength of evidence in favor of the treatment, 2 implementation-adjusted EVSI calculation methods are developed. These novel methods circumvent the need for analytical calculations, which were restricted to settings in which normality could be assumed. The first method developed in this article uses computationally demanding nested simulations, based on the definition of the implementation-adjusted EVSI. The second method is based on adapting the moment matching method, a recently developed efficient EVSI computation method, to adjust for imperfect implementation. The implementation-adjusted EVSI is then calculated with the 2 methods across 3 examples.

RESULTS

The maximum difference between the 2 methods is at most 6% in all examples. The efficient computation method is between 6 and 60 times faster than the nested simulation method in this case study and could be used in practice.

CONCLUSIONS

This article permits the calculation of an implementation-adjusted EVSI using realistic assumptions. The efficient estimation method is accurate and can estimate the implementation-adjusted EVSI in practice. By adapting standard EVSI estimation methods, adjustments for imperfect implementation can be made with the same computational cost as a standard EVSI analysis.

HIGHLIGHTS

Standard expected value of sample information (EVSI) analyses do not account for the fact that treatment implementation following research is often slow and incomplete, meaning they incorrectly capture the value of the study.Two methods, based on nested Monte Carlo sampling and the moment matching EVSI calculation method, are developed to adjust EVSI calculations for imperfect implementation when the speed and level of the implementation of a new treatment depends on the strength of evidence in favor of the treatment.The 2 methods we develop provide similar estimates for the implementation-adjusted EVSI.Our methods extend current EVSI calculation algorithms and thus require limited additional computational complexity.

Prioritizing Research in an Era of Personalized Medicine: The Potential Value of Unexplained Heterogeneity

Background

Clinical care is moving from a “one size fits all” approach to a setting in which treatment decisions are based on individual treatment response, needs, preferences, and risk. Research into personalized treatment strategies aims to discover currently unknown markers that identify individuals who would benefit from treatments that are nonoptimal at the population level. Before investing in research to identify these markers, it is important to assess whether such research has the potential to generate value. Thus, this article aims to develop a framework to prioritize research into the development of new personalized treatment strategies by creating a set of measures that assess the value of personalizing care based on a set of unknown patient characteristics.

Methods

Generalizing ideas from the value of heterogeneity framework, we demonstrate 3 measures that assess the value of developing personalized treatment strategies. The first measure identifies the potential value of personalizing medicine within a given disease area. The next 2 measures highlight specific research priorities and subgroup structures that would lead to improved patient outcomes from the personalization of treatment decisions.

Results

We graphically present the 3 measures to perform sensitivity analyses around the key drivers of value, in particular, the correlation between the individual treatment benefits across the available treatment options. We illustrate these 3 measures using a previously published decision model and discuss how they can direct research in personalized medicine.

Conclusion

We discuss 3 measures that form the basis of a novel framework to prioritize research into novel personalized treatment strategies. Our novel framework ensures that research targets personalized treatment strategies that have high potential to improve patient outcomes and health system efficiency.

Highlights

Modelling approaches for histology-independent cancer drugs to inform NICE appraisals: a systematic review and decision-framework

BACKGROUND

The first histology-independent marketing authorisation in Europe was granted in 2019. This was the first time that a cancer treatment was approved based on a common biomarker rather than the location in the body at which the tumour originated. This research aims to explore the implications for National Institute for Health and Care Excellence appraisals.

METHODS

Targeted reviews were undertaken to determine the type of evidence that is likely to be available at the point of marketing authorisation and the analyses required to support National Institute for Health and Care Excellence appraisals. Several challenges were identified concerning the design and conduct of trials for histology-independent products, the greater levels of heterogeneity within the licensed population and the use of surrogate end points. We identified approaches to address these challenges by reviewing key statistical literature that focuses on the design and analysis of histology-independent trials and by undertaking a systematic review to evaluate the use of response end points as surrogate outcomes for survival end points. We developed a decision framework to help to inform approval and research policies for histology-independent products. The framework explored the uncertainties and risks associated with different approval policies, including the role of further data collection, pricing schemes and stratified decision-making.

RESULTS

We found that the potential for heterogeneity in treatment effects, across tumour types or other characteristics, is likely to be a central issue for National Institute for Health and Care Excellence appraisals. Bayesian hierarchical methods may serve as a useful vehicle to assess the level of heterogeneity across tumours and to estimate the pooled treatment effects for each tumour, which can inform whether or not the assumption of homogeneity is reasonable. Our review suggests that response end points may not be reliable surrogates for survival end points. However, a surrogate-based modelling approach, which captures all relevant uncertainty, may be preferable to the use of immature survival data. Several additional sources of heterogeneity were identified as presenting potential challenges to National Institute for Health and Care Excellence appraisal, including the cost of testing, baseline risk, quality of life and routine management costs. We concluded that a range of alternative approaches will be required to address different sources of heterogeneity to support National Institute for Health and Care Excellence appraisals. An exemplar case study was developed to illustrate the nature of the assessments that may be required.

CONCLUSIONS

Adequately designed and analysed basket studies that assess the homogeneity of outcomes and allow borrowing of information across baskets, where appropriate, are recommended. Where there is evidence of heterogeneity in treatment effects and estimates of cost-effectiveness, consideration should be given to optimised recommendations. Routine presentation of the scale of the consequences of heterogeneity and decision uncertainty may provide an important additional approach to the assessments specified in the current National Institute for Health and Care Excellence methods guide.

FURTHER RESEARCH

Further exploration of Bayesian hierarchical methods could help to inform decision-makers on whether or not there is sufficient evidence of homogeneity to support pooled analyses. Further research is also required to determine the appropriate basis for apportioning genomic testing costs where there are multiple targets and to address the challenges of uncontrolled Phase II studies, including the role and use of surrogate end points.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Evidence Synthesis programme and will be published in full in Health Technology Assessment; Vol. 25, No. 76. See the NIHR Journals Library website for further project information.

Systematic Review of Cost-Effectiveness Models in Prostate Cancer: Exploring New Developments in Testing and Diagnosis

OBJECTIVES

Recent innovations in prostate cancer diagnosis include new biomarkers and more accurate biopsy methods. This study assesses the evidence base on cost-effectiveness of these developments (eg, Prostate Health Index and magnetic resonance imaging [MRI]-guided biopsy) and identifies areas of improvement for future cost-effectiveness models.

METHODS

A systematic review using the National Health Service Economic Evaluation Database, MEDLINE, Embase, Health Technology Assessment databases, National Institute for Health and Care Excellence guidelines, and United Kingdom National Screening Committee guidance was performed, between 2009 and 2021. Relevant data were extracted on study type, model inputs, modeling methods and cost-effectiveness conclusions, and results narratively synthesized.

RESULTS

A total of 22 model-based economic evaluations were included. A total of 11 compared the cost-effectiveness of new biomarkers to prostate-specific antigen testing alone and all found biomarkers to be cost saving. A total of 8 compared MRI-guided biopsy methods to transrectal ultrasound-guided methods and found MRI-guided methods to be most cost-effective. Newer detection methods showed a reduction in unnecessary biopsies and overtreatment. The most cost-effective follow-up strategy in men with a negative initial biopsy was uncertain. Many studies did not model for stage or grade of cancer, cancer progression, or the entire testing and treatment pathway. Few fully accounted for uncertainty.

CONCLUSIONS

This review brings together the cost-effectiveness literature for novel diagnostic methods in prostate cancer, showing that most studies have found new methods to be more cost-effective than standard of care. Several limitations of the models were identified, however, limiting the reliability of the results. Areas for further development include accurately modeling the impact of early diagnostic tests on long-term outcomes of prostate cancer and fully accounting for uncertainty.

Expected Value of Sample Information to Guide the Design of Group Sequential Clinical Trials

Introduction

Adaptive designs allow changes to an ongoing trial based on prespecified early examinations of accrued data. Opportunities are potentially being missed to incorporate health economic considerations into the design of these studies.

Methods

We describe how to estimate the expected value of sample information for group sequential design adaptive trials. We operationalize this approach in a hypothetical case study using data from a pilot trial. We report the expected value of sample information and expected net benefit of sampling results for 5 design options for the future full-scale trial including the fixed-sample-size design and the group sequential design using either the Pocock stopping rule or the O’Brien-Fleming stopping rule with 2 or 5 analyses. We considered 2 scenarios relating to 1) using the cost-effectiveness model with a traditional approach to the health economic analysis and 2) adjusting the cost-effectiveness analysis to incorporate the bias-adjusted maximum likelihood estimates of trial outcomes to account for the bias that can be generated in adaptive trials.

Results

The case study demonstrated that the methods developed could be successfully applied in practice. The results showed that the O’Brien-Fleming stopping rule with 2 analyses was the most efficient design with the highest expected net benefit of sampling in the case study.

Conclusions

Cost-effectiveness considerations are unavoidable in budget-constrained, publicly funded health care systems, and adaptive designs can provide an alternative to costly fixed-sample-size designs. We recommend that when planning a clinical trial, expected value of sample information methods be used to compare possible adaptive and nonadaptive trial designs, with appropriate adjustment, to help justify the choice of design characteristics and ensure the cost-effective use of research funding.

Highlights

A guide to value of information methods for prioritising research in health impact modelling

Health impact simulation models are used to predict how a proposed policy or scenario will affect population health outcomes. These models represent the typically-complex systems that describe how the scenarios affect exposures to risk factors for disease or injury (e.g. air pollution or physical inactivity), and how these risk factors are related to measures of population health (e.g. expected survival). These models are informed by multiple sources of data, and are subject to multiple sources of uncertainty. We want to describe which sources of uncertainty contribute most to uncertainty about the estimate or decision arising from the model. Furthermore, we want to decide where further research should be focused to obtain further data to reduce this uncertainty, and what form that research might take. This article presents a tutorial in the use of Value of Information methods for uncertainty analysis and research prioritisation in health impact simulation models. These methods are based on Bayesian decision-theoretic principles, and quantify the expected benefits from further information of different kinds. The expected value of partial perfect information about a parameter measures sensitivity of a decision or estimate to uncertainty about that parameter. The expected value of sample information represents the expected benefit from a specific proposed study to get better information about the parameter. The methods are applicable both to situationswhere the model is used to make a decision between alternative policies, and situations where the model is simply used to estimate a quantity (such as expected gains in survival under a scenario). This paper explains how to calculate and interpret the expected value of information in the context of a simple model describing the health impacts of air pollution from motorised transport. We provide a general-purpose R package and full code to reproduce the example analyses.

Using health technology assessment to set priority, inform target product profiles, and design clinical study for health innovation

Early health technology assessment (early HTA) is a useful tool in guiding the innovation development process in medical technology development. However, the application of early HTA is sub-optimal amongst research and development (R&D) communities due to several challenges. In this paper, we presented a case study of application of early HTA by drawing on the experience from a workshop conducted for the Singapore government's medical technology innovation agency. The framework developed can help maximise the chance of the newly developed technology being accepted and widely used. By providing step-by-step guidance, this work aims to translate early HTA into a practical tool and promote the application of early HTA amongst R&D communities.

Probabilistic threshold analysis by pairwise stochastic approximation for decision-making under uncertainty

The concept of probabilistic parameter threshold analysis has recently been introduced as a way of probabilistic sensitivity analysis for decision-making under uncertainty, in particular, for health economic evaluations which compare two or more alternative treatments with consideration of uncertainty on outcomes and costs. In this paper we formulate the probabilistic threshold analysis as a root-finding problem involving the conditional expectations, and propose a pairwise stochastic approximation algorithm to search for the threshold value below and above which the choice of conditionally optimal decision options changes. Numerical experiments for both a simple synthetic testcase and a chemotherapy Markov model illustrate the effectiveness of our proposed algorithm, without any need for accurate estimation or approximation of conditional expectations which the existing approaches rely upon. Moreover we introduce a new measure called decision switching probability for probabilistic sensitivity analysis in this paper.

Cancer diagnostic tools to aid decision-making in primary care: mixed-methods systematic reviews and cost-effectiveness analysis

BACKGROUND

Tools based on diagnostic prediction models are available to help general practitioners diagnose cancer. It is unclear whether or not tools expedite diagnosis or affect patient quality of life and/or survival.

OBJECTIVES

The objectives were to evaluate the evidence on the validation, clinical effectiveness, cost-effectiveness, and availability and use of cancer diagnostic tools in primary care.

METHODS

Two systematic reviews were conducted to examine the clinical effectiveness (review 1) and the development, validation and accuracy (review 2) of diagnostic prediction models for aiding general practitioners in cancer diagnosis. Bibliographic searches were conducted on MEDLINE, MEDLINE In-Process, EMBASE, Cochrane Library and Web of Science) in May 2017, with updated searches conducted in November 2018. A decision-analytic model explored the tools' clinical effectiveness and cost-effectiveness in colorectal cancer. The model compared patient outcomes and costs between strategies that included the use of the tools and those that did not, using the NHS perspective. We surveyed 4600 general practitioners in randomly selected UK practices to determine the proportions of general practices and general practitioners with access to, and using, cancer decision support tools. Association between access to these tools and practice-level cancer diagnostic indicators was explored.

RESULTS

Systematic review 1 - five studies, of different design and quality, reporting on three diagnostic tools, were included. We found no evidence that using the tools was associated with better outcomes. Systematic review 2 - 43 studies were included, reporting on prediction models, in various stages of development, for 14 cancer sites (including multiple cancers). Most studies relate to QCancer^® (ClinRisk Ltd, Leeds, UK) and risk assessment tools.

DECISION MODEL

In the absence of studies reporting their clinical outcomes, QCancer and risk assessment tools were evaluated against faecal immunochemical testing. A linked data approach was used, which translates diagnostic accuracy into time to diagnosis and treatment, and stage at diagnosis. Given the current lack of evidence, the model showed that the cost-effectiveness of diagnostic tools in colorectal cancer relies on demonstrating patient survival benefits. Sensitivity of faecal immunochemical testing and specificity of QCancer and risk assessment tools in a low-risk population were the key uncertain parameters.

SURVEY

Practitioner- and practice-level response rates were 10.3% (476/4600) and 23.3% (227/975), respectively. Cancer decision support tools were available in 83 out of 227 practices (36.6%, 95% confidence interval 30.3% to 43.1%), and were likely to be used in 38 out of 227 practices (16.7%, 95% confidence interval 12.1% to 22.2%). The mean 2-week-wait referral rate did not differ between practices that do and practices that do not have access to QCancer or risk assessment tools (mean difference of 1.8 referrals per 100,000 referrals, 95% confidence interval -6.7 to 10.3 referrals per 100,000 referrals).

LIMITATIONS

There is little good-quality evidence on the clinical effectiveness and cost-effectiveness of diagnostic tools. Many diagnostic prediction models are limited by a lack of external validation. There are limited data on current UK practice and clinical outcomes of diagnostic strategies, and there is no evidence on the quality-of-life outcomes of diagnostic results. The survey was limited by low response rates.

CONCLUSION

The evidence base on the tools is limited. Research on how general practitioners interact with the tools may help to identify barriers to implementation and uptake, and the potential for clinical effectiveness.

FUTURE WORK

Continued model validation is recommended, especially for risk assessment tools. Assessment of the tools' impact on time to diagnosis and treatment, stage at diagnosis, and health outcomes is also recommended, as is further work to understand how tools are used in general practitioner consultations.

STUDY REGISTRATION

This study is registered as PROSPERO CRD42017068373 and CRD42017068375.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology programme and will be published in full in Health Technology Assessment; Vol. 24, No. 66. See the NIHR Journals Library website for further project information.

Multilevel and Quasi Monte Carlo Methods for the Calculation of the Expected Value of Partial Perfect Information

The expected value of partial perfect information (EVPPI) provides an upper bound on the value of collecting further evidence on a set of inputs to a cost-effectiveness decision model. Standard Monte Carlo estimation of EVPPI is computationally expensive as it requires nested simulation. Alternatives based on regression approximations to the model have been developed but are not practicable when the number of uncertain parameters of interest is large and when parameter estimates are highly correlated. The error associated with the regression approximation is difficult to determine, while MC allows the bias and precision to be controlled. In this article, we explore the potential of quasi Monte Carlo (QMC) and multilevel Monte Carlo (MLMC) estimation to reduce the computational cost of estimating EVPPI by reducing the variance compared with MC while preserving accuracy. We also develop methods to apply QMC and MLMC to EVPPI, addressing particular challenges that arise where Markov chain Monte Carlo (MCMC) has been used to estimate input parameter distributions. We illustrate the methods using 2 examples: a simplified decision tree model for treatments for depression and a complex Markov model for treatments to prevent stroke in atrial fibrillation, both of which use MCMC inputs. We compare the performance of QMC and MLMC with MC and the approximation techniques of generalized additive model (GAM) regression, Gaussian process (GP) regression, and integrated nested Laplace approximations (INLA-GP). We found QMC and MLMC to offer substantial computational savings when parameter sets are large and correlated and when the EVPPI is large. We also found that GP and INLA-GP were biased in those situations, whereas GAM cannot estimate EVPPI for large parameter sets.

The Use and Misuse of Mathematical Modeling for Infectious Disease Policymaking: Lessons for the COVID-19 Pandemic

Mathematical modeling has played a prominent and necessary role in the current coronavirus disease 2019 (COVID-19) pandemic, with an increasing number of models being developed to track and project the spread of the disease, as well as major decisions being made based on the results of these studies. A proliferation of models, often diverging widely in their projections, has been accompanied by criticism of the validity of modeled analyses and uncertainty as to when and to what extent results can be trusted. Drawing on examples from COVID-19 and other infectious diseases of global importance, we review key limitations of mathematical modeling as a tool for interpreting empirical data and informing individual and public decision making. We present several approaches that have been used to strengthen the validity of inferences drawn from these analyses, approaches that will enable better decision making in the current COVID-19 crisis and beyond.

Prehospital continuous positive airway pressure for acute respiratory failure: the ACUTE feasibility RCT

BACKGROUND

Acute respiratory failure is a life-threatening emergency. Standard prehospital management involves controlled oxygen therapy. Continuous positive airway pressure is a potentially beneficial alternative treatment; however, it is uncertain whether or not this treatment could improve outcomes in NHS ambulance services.

OBJECTIVES

To assess the feasibility of a large-scale pragmatic trial and to update an existing economic model to determine cost-effectiveness and the value of further research.

DESIGN

(1) An open-label, individual patient randomised controlled external pilot trial. (2) Cost-effectiveness and value-of-information analyses, updating an existing economic model. (3) Ancillary substudies, comprising an acute respiratory failure incidence study, an acute respiratory failure diagnostic agreement study, clinicians perceptions of a continuous positive airway pressure mixed-methods study and an investigation of allocation concealment.

SETTING

Four West Midlands Ambulance Service hubs, recruiting between August 2017 and July 2018.

PARTICIPANTS

Adults with respiratory distress and peripheral oxygen saturations below the British Thoracic Society's target levels were included. Patients with limited potential to benefit from, or with contraindications to, continuous positive airway pressure were excluded.

INTERVENTIONS

Prehospital continuous positive airway pressure (O-Two system, O-Two Medical Technologies Inc., Brampton, ON, Canada) was compared with standard oxygen therapy, titrated to the British Thoracic Society's peripheral oxygen saturation targets. Interventions were provided in identical sealed boxes.

MAIN OUTCOME MEASURES

Feasibility objectives estimated the incidence of eligible patients, the proportion recruited and allocated to treatment appropriately, adherence to allocated treatment, and retention and data completeness. The primary clinical end point was 30-day mortality.

RESULTS

Seventy-seven patients were enrolled (target 120 patients), including seven patients with a diagnosis for which continuous positive airway pressure could be ineffective or harmful. Continuous positive airway pressure was fully delivered to 74% of participants (target 75%). There were no major protocol violations/non-compliances. Full data were available for all key outcomes (target ≥ 90%). Thirty-day mortality was 27.3%. Of the 21 deceased participants, 14 (68%) either did not have a respiratory condition or had ceiling-of-treatment decision implemented that excluded hospital non-invasive ventilation and critical care. The base-case economic evaluation indicated that standard oxygen therapy was probably cost-effective (incremental cost-effectiveness ratio £5685 per quality-adjusted life-year), but there was considerable uncertainty (population expected value of perfect information of £16.5M). Expected value of partial perfect information analyses indicated that effectiveness of prehospital continuous positive airway pressure was the only important variable. The incidence rate of acute respiratory failure was 17.4 (95% confidence interval 16.3 to 18.5) per 100,000 persons per year. There was moderate agreement between the primary prehospital and final hospital diagnoses (Gwet's AC1 coefficient 0.56, 95% confidence interval 0.43 to 0.69). Lack of hospital awareness of the Ambulance continuous positive airway pressure (CPAP): Use, Treatment Effect and economics (ACUTE) trial, limited time to complete trial training and a desire to provide continuous positive airway pressure treatment were highlighted as key challenges by participating clinicians.

LIMITATIONS

During week 10 of recruitment, the continuous positive airway pressure arm equipment boxes developed a 'rattle'. After repackaging and redistribution, no further concerns were noted. A total of 41.4% of ambulance service clinicians not participating in the ACUTE trial indicated a difference between the control and the intervention arm trial boxes (115/278); of these clinician 70.4% correctly identified box contents.

CONCLUSIONS

Recruitment rate was below target and feasibility was not demonstrated. The economic evaluation results suggested that a definitive trial could represent value for money. However, limited compliance with continuous positive airway pressure and difficulty in identifying patients who could benefit from continuous positive airway pressure indicate that prehospital continuous positive airway pressure is unlikely to materially reduce mortality.

FUTURE WORK

A definitive clinical effectiveness trial of continuous positive airway pressure in the NHS is not recommended.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN12048261.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 25, No. 7. See the NIHR Journals Library website for further project information.

A complex intervention to reduce avoidable hospital admissions in nursing homes: a research programme including the BHiRCH-NH pilot cluster RCT

Background

An unplanned hospital admission of a nursing home resident distresses the person, their family and nursing home staff, and is costly to the NHS. Improving health care in care homes, including early detection of residents’ health changes, may reduce hospital admissions. Previously, we identified four conditions associated with avoidable hospital admissions. We noted promising ‘within-home’ complex interventions including care pathways, knowledge and skills enhancement, and implementation support.

Objectives

Develop a complex intervention with implementation support [the Better Health in Residents in Care Homes with Nursing (BHiRCH-NH)] to improve early detection, assessment and treatment for the four conditions. Determine its impact on hospital admissions, test study procedures and acceptability of the intervention and implementation support, and indicate if a definitive trial was warranted.

Design

A Carer Reference Panel advised on the intervention, implementation support and study documentation, and engaged in data analysis and interpretation. In workstream 1, we developed a complex intervention to reduce rates of hospitalisation from nursing homes using mixed methods, including a rapid research review, semistructured interviews and consensus workshops. The complex intervention comprised care pathways, approaches to enhance staff knowledge and skills, implementation support and clarity regarding the role of family carers. In workstream 2, we tested the complex intervention and implementation support via two work packages. In work package 1, we conducted a feasibility study of the intervention, implementation support and study procedures in two nursing homes and refined the complex intervention to comprise the Stop and Watch Early Warning Tool (S&W), condition-specific care pathways and a structured framework for nurses to communicate with primary care. The final implementation support included identifying two Practice Development Champions (PDCs) in each intervention home, and supporting them with a training workshop, practice development support group, monthly coaching calls, handbooks and web-based resources. In work package 2, we undertook a cluster randomised controlled trial to pilot test the complex intervention for acceptability and a preliminary estimate of effect.

Setting

Fourteen nursing homes allocated to intervention and implementation support (n = 7) or treatment as usual (n = 7).

Participants

We recruited sufficient numbers of nursing homes (n = 14), staff (n = 148), family carers (n = 95) and residents (n = 245). Two nursing homes withdrew prior to the intervention starting.

Intervention

This ran from February to July 2018.

Data sources

Individual-level data on nursing home residents, their family carers and staff; system-level data using nursing home records; and process-level data comprising how the intervention was implemented. Data were collected on recruitment rates, consent and the numbers of family carers who wished to be involved in the residents’ care. Completeness of outcome measures and data collection and the return rate of questionnaires were assessed.

Results

The pilot trial showed no effects on hospitalisations or secondary outcomes. No home implemented the intervention tools as expected. Most staff endorsed the importance of early detection, assessment and treatment. Many reported that they ‘were already doing it’, using an early-warning tool; a detailed nursing assessment; or the situation, background, assessment, recommendation communication protocol. Three homes never used the S&W and four never used care pathways. Only 16 S&W forms and eight care pathways were completed. Care records revealed little use of the intervention principles. PDCs from five of six intervention homes attended the training workshop, following which they had variable engagement with implementation support. Progression criteria regarding recruitment and data collection were met: 70% of homes were retained, the proportion of missing data was < 20% and 80% of individual-level data were collected. Necessary rates of data collection, documentation completion and return over the 6-month study period were achieved. However, intervention tools were not fully adopted, suggesting they would not be sustainable outside the trial. Few hospitalisations for the four conditions suggest it an unsuitable primary outcome measure. Key cost components were estimated.

Limitations

The study homes may already have had effective approaches to early detection, assessment and treatment for acute health changes; consistent with government policy emphasising the need for enhanced health care in homes. Alternatively, the implementation support may not have been sufficiently potent.

Conclusion

A definitive trial is feasible, but the intervention is unlikely to be effective. Participant recruitment, retention, data collection and engagement with family carers can guide subsequent studies, including service evaluation and quality improvement methodologies.

Future work

Intervention research should be conducted in homes which need to enhance early detection, assessment and treatment. Interventions to reduce avoidable hospital admissions may be beneficial in residential care homes, as they are not required to employ nurses.

Trial registration

Current Controlled Trials ISRCTN74109734 and ISRCTN86811077.

Funding

This project was funded by the National Institute for Health Research (NIHR) Programme Grants for Applied Research programme and will be published in full in Programme Grants for Applied Research; Vol. 9, No. 2. See the NIHR Journals Library website for further project information.

Evaluating the relative contribution of data sources in a Bayesian analysis with the application of estimating the size of hard to reach populations

When using multiple data sources in an analysis, it is important to understand the influence of each data source on the analysis and the consistency of the data sources with each other and the model. We suggest the use of a retrospective value of information framework in order to address such concerns. Value of information methods can be computationally difficult. We illustrate the use of computational methods that allow these methods to be applied even in relatively complicated settings. In illustrating the proposed methods, we focus on an application in estimating the size of hard to reach populations. Specifically, we consider estimating the number of injection drug users in Ukraine by combining all available data sources spanning over half a decade and numerous sub-national areas in the Ukraine. This application is of interest to public health researchers as this hard to reach population that plays a large role in the spread of HIV. We apply a Bayesian hierarchical model and evaluate the contribution of each data source in terms of absolute influence, expected influence, and level of surprise. Finally we apply value of information methods to inform suggestions on future data collection.

Bayesian adaptive decision-theoretic designs for multi-arm multi-stage clinical trials

Multi-arm multi-stage clinical trials in which more than two drugs are simultaneously investigated provide gains over separate single- or two-arm trials. In this paper we propose a generic Bayesian adaptive decision-theoretic design for multi-arm multi-stage clinical trials with K (K≥2) arms. The basic idea is that after each stage a decision about continuation of the trial and accrual of patients for an additional stage is made on the basis of the expected reduction in loss. For this purpose, we define a loss function that incorporates the patient accrual costs as well as costs associated with an incorrect decision at the end of the trial. An attractive feature of our loss function is that its estimation is computationally undemanding, also when K > 2. We evaluate the frequentist operating characteristics for settings with a binary outcome and multiple experimental arms. We consider both the situation with and without a control arm. In a simulation study, we show that our design increases the probability of making a correct decision at the end of the trial as compared to nonadaptive designs and adaptive two-stage designs.

Linked Pharmacometric-Pharmacoeconomic Modeling and Simulation in Clinical Drug Development

Market access and pricing of pharmaceuticals are increasingly contingent on the ability to demonstrate comparative effectiveness and cost-effectiveness. As such, it is widely recognized that predictions of the economic potential of drug candidates in development could inform decisions across the product life cycle. This may be challenging when safety and efficacy profiles in terms of the relevant clinical outcomes are unknown or highly uncertain early in product development. Linking pharmacometrics and pharmacoeconomics, such that outputs from pharmacometric models serve as inputs to pharmacoeconomic models, may provide a framework for extrapolating from early-phase studies to predict economic outcomes and characterize decision uncertainty. This article reviews the published studies that have implemented this methodology and used simulation to inform drug development decisions and/or to optimize the use of drug treatments. Some of the key practical issues involved in linking pharmacometrics and pharmacoeconomics, including the choice of final outcome measures, methods of incorporating evidence on comparator treatments, approaches to handling multiple intermediate end points, approaches to quantifying uncertainty, and issues of model validation are also discussed. Finally, we have considered the potential barriers that may have limited the adoption of this methodology and suggest that closer alignment between the disciplines of clinical pharmacology, pharmacometrics, and pharmacoeconomics, may help to realize the potential benefits associated with linked pharmacometric-pharmacoeconomic modeling and simulation.

Comparing alternating pressure mattresses and high-specification foam mattresses to prevent pressure ulcers in high-risk patients: the PRESSURE 2 RCT

BACKGROUND

Pressure ulcers (PUs) are a burden to patients, carers and health-care providers. Specialist mattresses minimise the intensity and duration of pressure on vulnerable skin sites in at-risk patients.

PRIMARY OBJECTIVE

Time to developing a new PU of category ≥ 2 in patients using an alternating pressure mattress (APM) compared with a high-specification foam mattress (HSFM).

DESIGN

A multicentre, Phase III, open, prospective, planned as an adaptive double-triangular group sequential, parallel-group, randomised controlled trial with an a priori sample size of 2954 participants. Randomisation used minimisation (incorporating a random element).

SETTING

The trial was set in 42 secondary and community inpatient facilities in the UK.

PARTICIPANTS

Adult inpatients with evidence of acute illness and at a high risk of PU development.

INTERVENTIONS AND FOLLOW-UP

APM or HSFM - the treatment phase lasted a maximum of 60 days; the final 30 days were post-treatment follow-up.

MAIN OUTCOME MEASURES

Time to event.

RESULTS

From August 2013 to November 2016, 2029 participants were randomised to receive either APM (n = 1016) or HSFM (n = 1013). Primary end point - 30-day final follow-up: of the 2029 participants in the intention-to-treat population, 160 (7.9%) developed a new PU of category ≥ 2. There was insufficient evidence of a difference between groups for time to new PU of category ≥ 2 [Fine and Gray model HR 0.76, 95% confidence interval (CI) 0.56 to 1.04; exact p-value of 0.0890 and 2% absolute difference]. Treatment phase sensitivity analysis: 132 (6.5%) participants developed a new PU of category ≥ 2 between randomisation and end of treatment phase. There was a statistically significant difference in the treatment phase time-to-event sensitivity analysis (Fine and Gray model HR 0.66, 95% CI 0.46 to 0.93; p = 0.0176 and 2.6% absolute difference). Secondary end points - 30-day final follow-up: new PUs of category ≥ 1 developed in 350 (17.2%) participants, with no evidence of a difference between mattress groups in time to PU development, (Fine and Gray model HR 0.83, 95% CI 0.67 to 1.02; p-value = 0.0733 and absolute difference 3.1%). New PUs of category ≥ 3 developed in 32 (1.6%) participants with insufficient evidence of a difference between mattress groups in time to PU development (Fine and Gray model HR 0.81, 95% CI 0.40 to 1.62; p = 0.5530 and absolute difference 0.4%). Of the 145 pre-existing PUs of category 2, 89 (61.4%) healed - there was insufficient evidence of a difference in time to healing (Fine and Gray model HR 1.12, 95% CI 0.74 to 1.68; p = 0.6122 and absolute difference 2.9%). Health economics - the within-trial and long-term analysis showed APM to be cost-effective compared with HSFM; however, the difference in costs models are small and the quality-adjusted life-year gains are very small. There were no safety concerns. Blinded photography substudy - the reliability of central blinded review compared with clinical assessment for PUs of category ≥ 2 was 'very good' (kappa statistic 0.82, prevalence- and bias-adjusted kappa 0.82). Quality-of-life substudy - the Pressure Ulcer Quality of Life - Prevention (PU-QoL-P) instrument meets the established criteria for reliability, construct validity and responsiveness.

LIMITATIONS

A lower than anticipated event rate.

CONCLUSIONS

In acutely ill inpatients who are bedfast/chairfast and/or have a category 1 PU and/or localised skin pain, APMs confer a small treatment phase benefit that is diminished over time. Overall, the APM patient compliance, very low PU incidence rate observed and small differences between mattresses indicate the need for improved indicators for targeting of APMs and individualised decision-making. Decisions should take into account skin status, patient preferences (movement ability and rehabilitation needs) and the presence of factors that may be potentially modifiable through APM allocation, including being completely immobile, having nutritional deficits, lacking capacity and/or having altered skin/category 1 PU.

FUTURE WORK

Explore the relationship between mental capacity, levels of independent movement, repositioning and PU development. Explore 'what works for whom and in what circumstances'.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN01151335.

FUNDING

This project was funded by the National Institute for Health Research Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 23, No. 52. See the NIHR Journals Library website for further project information.

Optimal Allocation of Research Funds under a Budget Constraint

Purpose. Health economic evaluations that include the expected value of sample information support implementation decisions as well as decisions about further research. However, just as decision makers must consider portfolios of implementation spending, they must also identify the optimal portfolio of research investments. Methods. Under a fixed research budget, a decision maker determines which studies to fund; additional budget allocated to one study to increase the study sample size implies less budget available to collect information to reduce decision uncertainty in other implementation decisions. We employ a budget-constrained portfolio optimization framework in which the decisions are whether to invest in a study and at what sample size. The objective is to maximize the sum of the studies' population expected net benefit of sampling (ENBS). We show how to determine the optimal research portfolio and study-specific levels of investment. We demonstrate our framework with a stylized example to illustrate solution features and a real-world application using 6 published cost-effectiveness analyses. Results. Among the studies selected for nonzero investment, the optimal sample size occurs at the point at which the marginal population ENBS divided by the marginal cost of additional sampling is the same for all studies. Compared with standard ENBS optimization without a research budget constraint, optimal budget-constrained sample sizes are typically smaller but allow more studies to be funded. Conclusions. The budget constraint for research studies directly implies that the optimal sample size for additional research is not the point at which the ENBS is maximized for individual studies. A portfolio optimization approach can yield higher total ENBS. Ultimately, there is a maximum willingness to pay for incremental information that determines optimal sample sizes.

Economic Evaluation of Interventions to Address Opioid Misuse: A Systematic Review of Methods Used in Simulation Modeling Studies

OBJECTIVES

Several evidence-based interventions exist for people who misuse opioids, but there is limited guidance on optimal intervention selection. Economic evaluations using simulation modeling can guide the allocation of resources and help tackle the opioid crisis. This study reviews methods employed by economic evaluations using computer simulations to investigate the health and economic effects of interventions meant to address opioid misuse.

METHODS

We conducted a systematic mapping review of studies that used simulation modeling to support the economic evaluation of interventions targeting prevention, treatment, or management of opioid misuse or its direct consequences (ie, overdose). We searched 6 databases and extracted information on study population, interventions, costs, outcomes, and economic analysis and modeling approaches.

RESULTS

Eighteen studies met the inclusion criteria. All of the studies considered only one segment of the continuum of care. Of the studies, 13 evaluated medications for opioid use disorder, and 5 evaluated naloxone distribution programs to reduce overdose deaths. Most studies estimated incremental cost per quality-adjusted life-years and used health system and/or societal perspectives. Models were decision trees (n = 4), Markov (n = 10) or semi-Markov models (n = 3), and microsimulations (n = 1). All of the studies assessed parameter uncertainty though deterministic and/or probabilistic sensitivity analysis, 4 conducted formal calibration, only 2 assessed structural uncertainty, and only 1 conducted expected value of information analyses. Only 10 studies conducted validation.

CONCLUSIONS

Future economic evaluations should consider synergies between interventions and examine combinations of interventions to inform optimal policy response. They should also more consistently conduct model validation and assess the value of further research.

Computing the Expected Value of Sample Information Efficiently: Practical Guidance and Recommendations for Four Model-Based Methods

Value of information (VOI) analyses can help policy makers make informed decisions about whether to conduct and how to design future studies. Historically a computationally expensive method to compute the expected value of sample information (EVSI) restricted the use of VOI to simple decision models and study designs. Recently, 4 EVSI approximation methods have made such analyses more feasible and accessible. Members of the Collaborative Network for Value of Information (ConVOI) compared the inputs, the analyst's expertise and skills, and the software required for the 4 recently developed EVSI approximation methods. Our report provides practical guidance and recommendations to help inform the choice between the 4 efficient EVSI estimation methods. More specifically, this report provides: (1) a step-by-step guide to the methods' use, (2) the expertise and skills required to implement the methods, and (3) method recommendations based on the features of decision-analytic problems.

Sit–stand desks to reduce sedentary behaviour in 9- to 10-year-olds: the Stand Out in Class pilot cluster RCT

Background

Sedentary behaviour (sitting) is a highly prevalent negative health behaviour, with individuals of all ages exposed to environments that promote prolonged sitting. The school classroom represents an ideal setting for environmental change through the provision of sit–stand desks.

Objectives

The aim of this study was to undertake a pilot cluster randomised controlled trial of the introduction of sit–stand desks in primary school classrooms, to inform a definitive trial. Objectives included providing information on school and participant recruitment and retention, acceptability of the intervention, and outcome measures. A preliminary estimate of the intervention’s effectiveness on the proposed primary outcome (change in weekday sitting time) for inclusion in a definitive trial was calculated, along with a preliminary assessment of potential cost-effectiveness. A full process evaluation was also undertaken.

Design

A two-armed pilot cluster randomised controlled trial with economic and qualitative evaluations. Schools were randomised on a 1 : 1 basis to the intervention (n = 4) or control (n = 4) trial arms.

Setting

Primary schools in Bradford, West Yorkshire, UK.

Participants

Children in Year 5 (i.e. aged 9–10 years).

Intervention

Six sit–stand desks replaced three standard desks (sitting six children) in the intervention classrooms for 4.5 months. Teachers were encouraged to ensure that all pupils were exposed to the sit–stand desks for at least 1 hour per day, on average, using a rotation system. Schools assigned to the control arm continued with their usual practice.

Main outcome measures

Trial feasibility outcomes included school and participant recruitment and attrition, acceptability of the intervention, and acceptability of and compliance with the proposed outcome measures [including weekday sitting measured using activPAL™ (PAL Technologies Ltd, Glasgow, UK) accelerometers, physical activity, adiposity, blood pressure, cognitive function, musculoskeletal comfort, academic progress, engagement and behaviour].

Results

Thirty-three per cent of schools approached and 75% (n = 176) of eligible children took part. At the 7-month follow-up, retention rates were 100% for schools and 97% for children. Outcome measure completion rates ranged from 63% to 97%. A preliminary estimate of intervention effectiveness, from a weighted linear regression model (adjusting for baseline sitting time and wear time) revealed a mean difference in change in sitting of –30.6 minutes per day (95% confidence interval –56.42 to –4.84 minutes per day) between the intervention and control trial arms. The process evaluation revealed that the intervention, recruitment and evaluation procedures were acceptable to teachers and children, with the exception of minor issues around activPAL attachment. A preliminary within-trial economic analysis revealed no difference between intervention and control trial arms in health and education resource use or outcomes. Long-term modelling estimated an unadjusted incremental cost-effectiveness ratio of Stand Out in Class of £78,986 per quality-adjusted life-year gained.

Conclusion

This study has provided evidence of the acceptability and feasibility of the Stand Out in Class intervention and evaluation methods. Preliminary evidence suggests that the intervention may have a positive direction of effect on weekday sitting time, which warrants testing in a full cluster randomised controlled trial. Lessons learnt from this trial will inform the planning of a definitive trial.

Trial registration

Current Controlled Trials ISRCTN12915848.

Funding

This project was funded by the National Institute for Health Research (NIHR) Public Health Research programme and will be published in full in Public Health Research; Vol. 8, No. 8. See the NIHR Journals Library website for further project information.

Calculating the Expected Value of Sample Information in Practice: Considerations from 3 Case Studies

Background. Investing efficiently in future research to improve policy decisions is an important goal. Expected value of sample information (EVSI) can be used to select the specific design and sample size of a proposed study by assessing the benefit of a range of different studies. Estimating EVSI with the standard nested Monte Carlo algorithm has a notoriously high computational burden, especially when using a complex decision model or when optimizing over study sample sizes and designs. Recently, several more efficient EVSI approximation methods have been developed. However, these approximation methods have not been compared, and therefore their comparative performance across different examples has not been explored. Methods. We compared 4 EVSI methods using 3 previously published health economic models. The examples were chosen to represent a range of real-world contexts, including situations with multiple study outcomes, missing data, and data from an observational rather than a randomized study. The computational speed and accuracy of each method were compared. Results. In each example, the approximation methods took minutes or hours to achieve reasonably accurate EVSI estimates, whereas the traditional Monte Carlo method took weeks. Specific methods are particularly suited to problems where we wish to compare multiple proposed sample sizes, when the proposed sample size is large, or when the health economic model is computationally expensive. Conclusions. As all the evaluated methods gave estimates similar to those given by traditional Monte Carlo, we suggest that EVSI can now be efficiently computed with confidence in realistic examples. No systematically superior EVSI computation method exists as the properties of the different methods depend on the underlying health economic model, data generation process, and user expertise.

Value of Information Analytical Methods: Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force

The allocation of healthcare resources among competing priorities requires an assessment of the expected costs and health effects of investing resources in the activities and of the opportunity cost of the expenditure. To date, much effort has been devoted to assessing the expected costs and health effects, but there remains an important need to also reflect the consequences of uncertainty in resource allocation decisions and the value of further research to reduce uncertainty. Decision making with uncertainty may turn out to be suboptimal, resulting in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to better inform resource decisions. This value can be quantified using value of information (VOI) analysis. This report from the ISPOR VOI Task Force describes methods for computing 4 VOI measures: the expected value of perfect information, expected value of partial perfect information (EVPPI), expected value of sample information (EVSI), and expected net benefit of sampling (ENBS). Several methods exist for computing EVPPI and EVSI, and this report provides guidance on selecting the most appropriate method based on the features of the decision problem. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing VOI analyses. The software needed to compute VOI is discussed, and areas for future research are highlighted.

"Stick or Twist?" Negotiating Price and Data in an Era of Conditional Approval

BACKGROUND

Changes in the regulatory context enable faster approval of transformative medicines. They also lead to health technology assessment (HTA) agencies having to make decisions with less evidence. In response, HTA agencies have also initiated forms of conditional approval. When the evidence base for a new oncology treatment leaves substantial uncertainty, the new Cancer Drugs Fund allows the National Institute for Heath and Care Excellence to give the manufacturer two options: (1) offer a low price based on conservative assumptions and obtain immediate approval ("stick") or (2) wait until the evidence base has further matured before finalizing a potentially higher agreed price ("twist").

OBJECTIVES

The purpose of this article is to explain how, using the theoretical framework of the expected value of sample information, simulation methods can help inform a manufacturer's decisions when faced with the option to stick or twist.

METHODS

We first summarize a general model to help frame the manufacturer's negotiating strategy. We then use a motivating case study, based on a hypothetical immunotherapy, to illustrate how manufacturers can use simulation methods to robustly characterize the uncertainty inherent to further data collection and incorporate this uncertainty within their decision making.

RESULTS

Our approach allows us to estimate the commercial value of generating additional data (the difference between the estimated net present value of stick and twist). We test the sensitivity of the results to different assumptions via scenario analyses.

CONCLUSIONS

This article shows that simulation methods can be used to help pharmaceutical managers make informed strategic decisions in contexts of uncertainty.

Influence of Modeling Choices on Value of Information Analysis: An Empirical Analysis from a Real-World Experiment

BACKGROUND

Value of information (VOI) analysis often requires modeling to characterize and propagate uncertainty. In collaboration with a cancer clinical trial group, we integrated a VOI approach to assessing trial proposals.

OBJECTIVE

This paper aims to explore the impact of modeling choices on VOI results and to share lessons learned from the experience.

METHODS

After selecting two proposals (A: phase III, breast cancer; B: phase II, pancreatic cancer) for in-depth evaluations, we categorized key modeling choices relevant to trial decision makers (characterizing uncertainty of efficacy, evidence thresholds to change clinical practice, and sample size) and modelers (cycle length, survival distribution, simulation runs, and other choices). Using a $150,000 per quality-adjusted life-year (QALY) threshold, we calculated the patient-level expected value of sample information (EVSI) for each proposal and examined whether each modeling choice led to relative change of more than 10% from the averaged base-case estimate. We separately analyzed the impact of the effective time horizon.

RESULTS

The base-case EVSI was $118,300 for Proposal A and $22,200 for Proposal B per patient. Characterizing uncertainty of efficacy was the most important choice in both proposals (e.g. Proposal A: $118,300 using historical data vs. $348,300 using expert survey), followed by the sample size and the choice of survival distribution. The assumed effective time horizon also had a substantial impact on the population-level EVSI.

CONCLUSIONS

Modeling choices can have a substantial impact on VOI. Therefore, it is important for groups working to incorporate VOI into research prioritization to adhere to best practices, be clear in their reporting and justification for modeling choices, and to work closely with the relevant decision makers, with particular attention to modeling choices.

A Framework to Prioritise Health Research Proposals for Funding: Integrating Value for Money

When making funding decisions, research organisations largely consider the merits (e.g. scientific rigour and feasibility) of submitted research proposals; yet, there is often little or no reference to their value for money. This may be attributed to the challenges of assessing and integrating value of research into existing research prioritisation processes. We propose a framework that considers both the merits of research and its value for money to guide health research funding decisions. A practical framework is developed based on current processes followed by funding organizations for assessing investigator-initiated research proposals, and analytical methods for evaluating the expected value of research. We apply the analytical methods to estimate the expected return on investment of two real-world grant applications. The framework comprises four sequential steps: (1) initial screening of applications for eligibility and completeness; (2) merit assessment of eligible proposals; (3) estimating the expected value of research for the shortlisted proposals that pass the first two steps and ranking of proposals based on return on investment; and (4) selecting research proposals for funding. We demonstrate how the expected value for money can be efficiently estimated using certain information provided in funding applications. The proposed framework integrates value-for-money assessment into the existing research prioritisation processes. Considering value for money to inform research funding decisions is vital to achieve efficient utilisation of research budgets and maximise returns on research investments.

Antidepressant treatment with sertraline for adults with depressive symptoms in primary care: the PANDA research programme including RCT

Background

Despite a growing number of prescriptions for antidepressants (over 70 million in 2018), there is uncertainty about when people with depression might benefit from antidepressant medication and concern that antidepressants are prescribed unnecessarily.

Objectives

The main objective of the PANDA (What are the indications for Prescribing ANtiDepressAnts that will lead to a clinical benefit?) research programme was to provide more guidance about when antidepressants are likely to benefit people with depression. We aimed to estimate the minimal clinically important difference for commonly used self-administered scales for depression and anxiety, and to understand more about how patients respond to such assessments. We carried out an observational study of patients with depressive symptoms and a placebo-controlled randomised controlled trial of sertraline versus placebo to estimate the treatment effect in UK primary care. The hypothesis was that the severity and duration of symptoms were related to treatment response.

Design

The programme consisted of three phases. The first phase relied on the secondary analysis of existing data extracted from published trials. The second phase was the PANDA cohort study of patients with depressive symptoms who presented to primary care and were followed up 2, 4 and 6 weeks after a baseline assessment. Both quantitative and qualitative methods were used in the analysis. The third phase was a multicentre randomised placebo-controlled double-blind trial of sertraline versus placebo in patients presenting to primary care with depressive symptoms.

Setting

UK primary care in Bristol, London, Liverpool and York.

Participants

Patients aged 18–74 years who were experiencing depressive symptoms in primary care. Eligibility for the PANDA randomised controlled trial included that there was uncertainty about the benefits about treatment with an antidepressant.

Interventions

In the PANDA randomised controlled trial, patients were individually randomised to 100 mg daily of sertraline or an identical placebo. The PANDA cohort study was an observational study.

Main outcome measures

Depressive symptoms measured using the Patient Health Questionnaire were the primary outcome for the randomised controlled trial. Other outcomes included anxiety symptoms using the Generalised Anxiety Disorder-7; depressive symptoms using the Beck Depression Inventory, version 2; health-related quality of life; self-reported improvement; and cost-effectiveness.

Results

The secondary analysis of existing randomised controlled trials [GENetic and clinical Predictors Of treatment response in Depression (GenPod), TREAting Depression with physical activity (TREAD) and Clinical effectiveness and cost-effectiveness of cognitive Behavioural Therapy as an adjunct to pharmacotherapy for treatment-resistant depression in primary care (CoBalT)] found evidence that the minimal clinically important difference increased as the initial severity of depressive symptoms rose. Our estimates of minimal clinically important difference were a 17% and 18% reduction in Beck Depression Inventory scores for GenPod and TREAD, respectively. In CoBalT, a 32% reduction corresponded to the minimal clinically important difference but the participants in this study had depression that had not responded to antidepressants. In the PANDA study cohort, and from our analyses in existing data, we found that the minimal clinically important difference varies considerably with the initial severity of depressive and anxiety symptoms. Expressing the minimal clinically important difference as a percentage reduction reduces this variation at higher scores, but at low scores the percentage reduction increased substantially. The results from the qualitative studies pointed out many limitations of the Patient Health Questionnaire-9 items in assessing change and recovery from depression. In the PANDA randomised controlled trial, there was no evidence that sertraline resulted in a reduction in depressive symptoms within 6 weeks of randomisation, but there was some evidence of a reduction by 12 weeks. However, sertraline led to a reduction in anxiety symptoms, an improvement of mental health-related quality of life and an increased likelihood of reporting improvement. The mean Patient Health Questionnaire-9 items score at 6 weeks was 7.98 (standard deviation 5.63) in the sertraline group and 8.76 (standard deviation 5.86) in the placebo group (5% relative reduction, 95% confidence interval –7% to 15%; p = 0.41). Of the secondary outcomes, there was strong evidence that sertraline reduced anxiety symptoms (Generalised Anxiety Disorder-7 score reduced by 17% (95% confidence interval 9% to 25%; p = 0.00005). Sertraline had a high probability (> 90%) of being cost-effective at 12 weeks. The PANDA randomised controlled trial found no evidence that treatment response or cost-effectiveness was related to severity or duration of depressive symptoms. The minimal clinically important difference estimates suggested that sertraline’s effect on anxiety, but not on depression, was likely to be clinically important.

Limitations

The results from the randomised controlled trial and the estimates of minimal clinically important difference were not sufficiently precise to provide specific clinical guidance for individuals. We had low power in testing whether or not initial severity and duration of depressive symptoms are related to treatment response.

Conclusions

The results of the trial support the use of sertraline and probably other selective serotonin reuptake inhibitors because of their action in reducing anxiety symptoms and the likelihood of longer-term benefit on depressive symptoms. Sertraline could be prescribed for anxiety symptoms that commonly occur with depression and many patients will experience a clinical benefit. The Patient Health Questionnaire-9 items and similar self-administered scales should not be used on their own to assess clinical outcome, but should be supplemented with further clinical assessment.

Future work

We need to examine the longer-term effects of antidepressant treatment. We need more precise estimates of the treatment effects and minimal clinically important difference at different severities to provide more specific guidance for individuals. However, the methods we have developed provide an approach towards providing such detailed guidance.

Trial registration

Current Controlled Trials ISRCTN84544741 and EudraCT number 2013-003440-22.

Funding

This project was funded by the National Institute for Health Research (NIHR) Programme Grants for Applied Research programme and will be published in full in Programme Grants for Applied Research; Vol. 7, No. 10. See the NIHR Journals Library website for further project information.

Accumulating evidence in ecology: Once is not enough

Many published studies in ecological science are viewed as stand-alone investigations that purport to provide new insights into how ecological systems behave based on single analyses. But it is rare for results of single studies to provide definitive results, as evidenced in current discussions of the "reproducibility crisis" in science. The key step in science is the comparison of hypothesis-based predictions with observations, where the predictions are typically generated by hypothesis-specific models. Repeating this step allows us to gain confidence in the predictive ability of a model, and its corresponding hypothesis, and thus to accumulate evidence and eventually knowledge. This accumulation may occur via an ad hoc approach, via meta-analyses, or via a more systematic approach based on the anticipated evolution of an information state. We argue the merits of this latter approach, provide an example, and discuss implications for designing sequences of studies focused on a particular question. We conclude by discussing current data collection programs that are preadapted to use this approach and argue that expanded use would increase the rate of learning in ecology, as well as our confidence in what is learned.

Directly Acting Oral Anticoagulants for the Prevention of Stroke in Atrial Fibrillation in England and Wales: Cost-Effectiveness Model and Value of Information Analysis

Objectives. Determine the optimal, licensed, first-line anticoagulant for prevention of ischemic stroke in patients with non-valvular atrial fibrillation (AF) in England and Wales from the UK National Health Service (NHS) perspective and estimate value to decision making of further research. Methods. We developed a cost-effectiveness model to compare warfarin (international normalized ratio target range 2-3) with directly acting (or non-vitamin K antagonist) oral anticoagulants (DOACs) apixaban 5 mg, dabigatran 150 mg, edoxaban 60 mg, and rivaroxaban 20 mg, over 30 years post treatment initiation. In addition to death, the 17-state Markov model included the events stroke, bleed, myocardial infarction, and intracranial hemorrhage. Input parameters were informed by systematic literature reviews and network meta-analysis. Expected value of perfect information (EVPI) and expected value of partial perfect information (EVPPI) were estimated to provide an upper bound on value of further research. Results. At willingness-to-pay threshold £20,000, all DOACs have positive expected incremental net benefit compared to warfarin, suggesting they are likely cost-effective. Apixaban has highest expected incremental net benefit (£7533), followed by dabigatran (£6365), rivaroxaban (£5279), and edoxaban (£5212). There was considerable uncertainty as to the optimal DOAC, with the probability apixaban has highest net benefit only 60%. Total estimated population EVPI was £17.94 million (17.85 million, 18.03 million), with relative effect between apixaban versus dabigatran making the largest contribution with EVPPI of £7.95 million (7.66 million, 8.24 million). Conclusions. At willingness-to-pay threshold £20,000, all DOACs have higher expected net benefit than warfarin but there is considerable uncertainty between the DOACs. Apixaban had the highest expected net benefit and greatest probability of having highest net benefit, but there is considerable uncertainty between DOACs. A head-to-head apixaban versus dabigatran trial may be of value.

The Curve of Optimal Sample Size (COSS): A Graphical Representation of the Optimal Sample Size from a Value of Information Analysis

Value of information (VOI) analysis quantifies the opportunity cost associated with decision uncertainty, and thus informs the value of collecting further information to avoid this cost. VOI can inform study design, optimal sample size selection, and research prioritization. Recent methodological advances have reduced the computational burden of conducting VOI analysis and have made it easier to evaluate the expected value of sample information, the expected net benefit of sampling, and the optimal sample size of a study design ([Formula: see text]). The volume of VOI analyses being published is increasing, and there is now a need for VOI studies to conduct sensitivity analyses on VOI-specific parameters. In this practical application, we introduce the curve of optimal sample size (COSS), which is a graphical representation of [Formula: see text] over a range of willingness-to-pay thresholds and VOI parameters (example data and R code are provided). In a single figure, the COSS presents summary data for decision makers to determine the sample size that optimizes research funding given their operating characteristics. The COSS also presents variation in the optimal sample size given variability or uncertainty in VOI parameters. The COSS represents an efficient and additional approach for summarizing results from a VOI analysis.