Exploring the research decision space: the expected value of information for sequential research designs

Methods to Quantify the Importance of Parameters for Model Updating and Distributional Adaptation

PURPOSE

Decision models are time-consuming to develop; therefore, adapting previously developed models for new purposes may be advantageous. We provide methods to prioritize efforts to 1) update parameter values in existing models and 2) adapt existing models for distributional cost-effectiveness analysis (DCEA).

METHODS

Methods exist to assess the influence of different input parameters on the results of a decision models, including value of information (VOI) and 1-way sensitivity analysis (OWSA). We apply 1) VOI to prioritize searches for additional information to update parameter values and 2) OWSA to prioritize searches for parameters that may vary by socioeconomic characteristics. We highlight the assumptions required and propose metrics that quantify the extent to which parameters in a model have been updated or adapted. We provide R code to quickly carry out the analysis given inputs from a probabilistic sensitivity analysis (PSA) and demonstrate our methods using an oncology case study.

RESULTS

In our case study, updating 2 of 21 probabilistic model parameters addressed 71.5% of the total VOI and updating 3 addressed approximately 100% of the uncertainty. Our proposed approach suggests that these are the 3 parameters that should be prioritized. For model adaptation for DCEA, 46.3% of the total OWSA variation came from a single parameter, while the top 10 input parameters were found to account for more than 95% of the total variation, suggesting efforts should be aimed toward these.

CONCLUSIONS

These methods offer a systematic approach to guide research efforts in updating models with new data or adapting models to undertake DCEA. The case study demonstrated only very small gains from updating more than 3 parameters or adapting more than 10 parameters.

HIGHLIGHTS

It can require considerable analyst time to search for evidence to update a model or to adapt a model to take account of equity concerns.In this article, we provide a quantitative method to prioritze parameters to 1) update existing models to reflect potential new evidence and 2) adapt existing models to estimate distributional outcomes.We define metrics that quantify the extent to which the parameters in a model have been updated or adapted.We provide R code that can quickly rank parameter importance and calculate quality metrics using only the results of a standard probabilistic sensitivity analysis.

Expected Value of Sample Information to Guide the Design of Group Sequential Clinical Trials

Introduction

Adaptive designs allow changes to an ongoing trial based on prespecified early examinations of accrued data. Opportunities are potentially being missed to incorporate health economic considerations into the design of these studies.

Methods

We describe how to estimate the expected value of sample information for group sequential design adaptive trials. We operationalize this approach in a hypothetical case study using data from a pilot trial. We report the expected value of sample information and expected net benefit of sampling results for 5 design options for the future full-scale trial including the fixed-sample-size design and the group sequential design using either the Pocock stopping rule or the O’Brien-Fleming stopping rule with 2 or 5 analyses. We considered 2 scenarios relating to 1) using the cost-effectiveness model with a traditional approach to the health economic analysis and 2) adjusting the cost-effectiveness analysis to incorporate the bias-adjusted maximum likelihood estimates of trial outcomes to account for the bias that can be generated in adaptive trials.

Results

The case study demonstrated that the methods developed could be successfully applied in practice. The results showed that the O’Brien-Fleming stopping rule with 2 analyses was the most efficient design with the highest expected net benefit of sampling in the case study.

Conclusions

Cost-effectiveness considerations are unavoidable in budget-constrained, publicly funded health care systems, and adaptive designs can provide an alternative to costly fixed-sample-size designs. We recommend that when planning a clinical trial, expected value of sample information methods be used to compare possible adaptive and nonadaptive trial designs, with appropriate adjustment, to help justify the choice of design characteristics and ensure the cost-effective use of research funding.

Highlights

Behavioural activation therapy for post-stroke depression: the BEADS feasibility RCT

BACKGROUND

There is currently insufficient evidence for the clinical effectiveness and cost-effectiveness of psychological therapies for post-stroke depression.

OBJECTIVE

To evaluate the feasibility of undertaking a definitive trial to evaluate the clinical effectiveness and cost-effectiveness of behavioural activation (BA) compared with usual stroke care for treating post-stroke depression.

DESIGN

Parallel-group, feasibility, multicentre, randomised controlled trial with nested qualitative research and a health economic evaluation.

SETTING

Acute and community stroke services in three sites in England.

PARTICIPANTS

Community-dwelling adults 3 months to 5 years post stroke who are depressed, as determined by the Patient Health Questionnaire-9 (PHQ-9) or the Visual Analogue Mood Scales 'Sad' item. Exclusions: patients who are blind and/or deaf, have dementia, are unable to communicate in English, do not have mental capacity to consent, are receiving treatment for depression at the time of stroke onset or are currently receiving psychological intervention.

RANDOMISATION AND BLINDING

Participants were randomised (1 : 1 ratio) to BA or usual stroke care. Randomisation was conducted using a computer-generated list with random permuted blocks of varying sizes, stratified by site. Participants and therapists were aware of the allocation, but outcome assessors were blind.

INTERVENTIONS

The intervention arm received up to 15 sessions of BA over 4 months. BA aims to improve mood by increasing people's level of enjoyable or valued activities. The control arm received usual care only.

MAIN OUTCOME MEASURES

Primary feasibility outcomes concerned feasibility of recruitment to the main trial, acceptability of research procedures and measures, appropriateness of baseline and outcome measures, retention of participants and potential value of conducting the definitive trial. Secondary feasibility outcomes concerned the delivery of the intervention. The primary clinical outcome 6 months post randomisation was the PHQ-9. Secondary clinical outcomes were Stroke Aphasic Depression Questionnaire - Hospital version, Nottingham Leisure Questionnaire, Nottingham Extended Activities of Daily Living, Carer Strain Index, EuroQol-5 Dimensions, five-level version and health-care resource use questionnaire.

RESULTS

Forty-eight participants were recruited in 27 centre-months of recruitment, at a recruitment rate of 1.8 participants per centre per month. The 25 participants randomised to receive BA attended a mean of 8.5 therapy sessions [standard deviation (SD) 4.4 therapy sessions]; 23 participants were allocated to usual care. Outcome assessments were completed by 39 (81%) participants (BA, n = 18; usual care, n = 21). Mean PHQ-9 scores at 6-month follow-up were 10.1 points (SD 6.9 points) and 14.4 points (SD 5.1 points) in the BA and control groups, respectively, a difference of -3.8 (95% confidence interval -6.9 to -0.6) after adjusting for baseline PHQ-9 score and centre, representing a reduction in depression in the BA arm. Therapy was delivered as intended. BA was acceptable to participants, carers and therapists. Value-of-information analysis indicates that the benefits of conducting a definitive trial would be likely to outweigh the costs. It is estimated that a sample size of between 580 and 623 participants would be needed for a definitive trial.

LIMITATIONS

Target recruitment was not achieved, although we identified methods to improve recruitment.

CONCLUSIONS

The Behavioural Activation Therapy for Depression after Stroke trial was feasible with regard to the majority of outcomes. The outstanding issue is whether or not a sufficient number of participants could be recruited within a reasonable time frame for a definitive trial. Future work is required to identify whether or not there are sufficient sites that are able to deliver the services required for a definitive trial.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN12715175.

FUNDING

This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 23, No. 47. See the NIHR Journals Library website for further project information.

Optimal Allocation of Research Funds under a Budget Constraint

Purpose. Health economic evaluations that include the expected value of sample information support implementation decisions as well as decisions about further research. However, just as decision makers must consider portfolios of implementation spending, they must also identify the optimal portfolio of research investments. Methods. Under a fixed research budget, a decision maker determines which studies to fund; additional budget allocated to one study to increase the study sample size implies less budget available to collect information to reduce decision uncertainty in other implementation decisions. We employ a budget-constrained portfolio optimization framework in which the decisions are whether to invest in a study and at what sample size. The objective is to maximize the sum of the studies' population expected net benefit of sampling (ENBS). We show how to determine the optimal research portfolio and study-specific levels of investment. We demonstrate our framework with a stylized example to illustrate solution features and a real-world application using 6 published cost-effectiveness analyses. Results. Among the studies selected for nonzero investment, the optimal sample size occurs at the point at which the marginal population ENBS divided by the marginal cost of additional sampling is the same for all studies. Compared with standard ENBS optimization without a research budget constraint, optimal budget-constrained sample sizes are typically smaller but allow more studies to be funded. Conclusions. The budget constraint for research studies directly implies that the optimal sample size for additional research is not the point at which the ENBS is maximized for individual studies. A portfolio optimization approach can yield higher total ENBS. Ultimately, there is a maximum willingness to pay for incremental information that determines optimal sample sizes.

Computerised speech and language therapy or attention control added to usual care for people with long-term post-stroke aphasia: the Big CACTUS three-arm RCT

BACKGROUND

People with aphasia may improve their communication with speech and language therapy many months/years after stroke. However, NHS speech and language therapy reduces in availability over time post stroke.

OBJECTIVE

This trial evaluated the clinical effectiveness and cost-effectiveness of self-managed computerised speech and language therapy to provide additional therapy.

DESIGN

A pragmatic, superiority, single-blind, parallel-group, individually randomised (stratified block randomisation, stratified by word-finding severity and site) adjunct trial.

SETTING

Twenty-one UK NHS speech and language therapy departments.

PARTICIPANTS

People with post-stroke aphasia (diagnosed by a speech and language therapist) with long-standing (> 4 months) word-finding difficulties.

INTERVENTIONS

The groups were (1) usual care; (2) daily self-managed computerised word-finding therapy tailored by speech and language therapists and supported by volunteers/speech and language therapy assistants for 6 months plus usual care (computerised speech and language therapy); and (3) activity/attention control (completion of puzzles and receipt of telephone calls from a researcher for 6 months) plus usual care.

MAIN OUTCOME MEASURES

Co-primary outcomes - change in ability to find treated words of personal relevance in a bespoke naming test (impairment) and change in functional communication in conversation rated on the activity scale of the Therapy Outcome Measures (activity) 6 months after randomisation. A key secondary outcome was participant-rated perception of communication and quality of life using the Communication Outcomes After Stroke questionnaire at 6 months. Outcomes were assessed by speech and language therapists using standardised procedures. Cost-effectiveness was estimated using treatment costs and an accessible EuroQol-5 Dimensions, five-level version, measuring quality-adjusted life-years.

RESULTS

A total of 818 patients were assessed for eligibility and 278 participants were randomised between October 2014 and August 2016. A total of 240 participants (86 usual care, 83 computerised speech and language therapy, 71 attention control) contributed to modified intention-to-treat analysis at 6 months. The mean improvements in word-finding were 1.1% (standard deviation 11.2%) for usual care, 16.4% (standard deviation 15.3%) for computerised speech and language therapy and 2.4% (standard deviation 8.8%) for attention control. Computerised speech and language therapy improved word-finding 16.2% more than usual care did (95% confidence interval 12.7% to 19.6%; p < 0.0001) and 14.4% more than attention control did (95% confidence interval 10.8% to 18.1%). Most of this effect was maintained at 12 months (n = 219); the mean differences in change in word-finding score were 12.7% (95% confidence interval 8.7% to 16.7%) higher in the computerised speech and language therapy group (n = 74) than in the usual-care group (n = 84) and 9.3% (95% confidence interval 4.8% to 13.7%) higher in the computerised speech and language therapy group than in the attention control group (n = 61). Computerised speech and language therapy did not show significant improvements on the Therapy Outcome Measures or Communication Outcomes After Stroke scale compared with usual care or attention control. Primary cost-effectiveness analysis estimated an incremental cost per participant of £732.73 (95% credible interval £674.23 to £798.05). The incremental quality-adjusted life-year gain was 0.017 for computerised speech and language therapy compared with usual care, but its direction was uncertain (95% credible interval -0.05 to 0.10), resulting in an incremental cost-effectiveness ratio of £42,686 per quality-adjusted life-year gained. For mild and moderate word-finding difficulty subgroups, incremental cost-effectiveness ratios were £22,371 and £28,898 per quality-adjusted life-year gained, respectively, for computerised speech and language therapy compared with usual care.

LIMITATIONS

This trial excluded non-English-language speakers, the accessible EuroQol-5 Dimensions, five-level version, was not validated and the measurement of attention control fidelity was limited.

CONCLUSIONS

Computerised speech and language therapy enabled additional self-managed speech and language therapy, contributing to significant improvement in finding personally relevant words (as specifically targeted by computerised speech and language therapy) long term post stroke. Gains did not lead to improvements in conversation or quality of life. Cost-effectiveness is uncertain owing to uncertainty around the quality-adjusted life-year gain, but computerised speech and language therapy may be more cost-effective for participants with mild and moderate word-finding difficulties. Exploring ways of helping people with aphasia to use new words in functional communication contexts is a priority.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN68798818.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 24, No. 19. See the NIHR Journals Library website for further project information. The Tavistock Trust for Aphasia provided additional support to enable people in the control groups to experience the intervention after the trial had ended.

Value of Information Analytical Methods: Report 2 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force

The allocation of healthcare resources among competing priorities requires an assessment of the expected costs and health effects of investing resources in the activities and of the opportunity cost of the expenditure. To date, much effort has been devoted to assessing the expected costs and health effects, but there remains an important need to also reflect the consequences of uncertainty in resource allocation decisions and the value of further research to reduce uncertainty. Decision making with uncertainty may turn out to be suboptimal, resulting in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to better inform resource decisions. This value can be quantified using value of information (VOI) analysis. This report from the ISPOR VOI Task Force describes methods for computing 4 VOI measures: the expected value of perfect information, expected value of partial perfect information (EVPPI), expected value of sample information (EVSI), and expected net benefit of sampling (ENBS). Several methods exist for computing EVPPI and EVSI, and this report provides guidance on selecting the most appropriate method based on the features of the decision problem. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing VOI analyses. The software needed to compute VOI is discussed, and areas for future research are highlighted.

Value of Information Analysis for Research Decisions-An Introduction: Report 1 of the ISPOR Value of Information Analysis Emerging Good Practices Task Force

Healthcare resource allocation decisions made under conditions of uncertainty may turn out to be suboptimal. In a resource constrained system in which there is a fixed budget, these suboptimal decisions will result in health loss. Consequently, there may be value in reducing uncertainty, through the collection of new evidence, to make better resource allocation decisions. This value can be quantified using a value of information (VOI) analysis. This report, from the ISPOR VOI Task Force, introduces VOI analysis, defines key concepts and terminology, and outlines the role of VOI for supporting decision making, including the steps involved in undertaking and interpreting VOI analyses. The report is specifically aimed at those tasked with making decisions about the adoption of healthcare or the funding of healthcare research. The report provides a number of recommendations for good practice when planning, undertaking, or reviewing the results of VOI analyses.

A Review of Clinical Trials With an Adaptive Design and Health Economic Analysis

OBJECTIVE

An adaptive design uses data collected as a clinical trial progresses to inform modifications to the trial. Hence, adaptive designs and health economics aim to facilitate efficient and accurate decision making. Nevertheless, it is unclear whether the methods are considered together in the design, analysis, and reporting of trials. This review aims to establish how health economic outcomes are used in the design, analysis, and reporting of adaptive designs.

METHODS

Registered and published trials up to August 2016 with an adaptive design and health economic analysis were identified. The use of health economics in the design, analysis, and reporting was assessed. Summary statistics are presented and recommendations formed based on the research team's experiences and a practical interpretation of the results.

RESULTS

Thirty-seven trials with an adaptive design and health economic analysis were identified. It was not clear whether the health economic analysis accounted for the adaptive design in 17/37 trials where this was thought necessary, nor whether health economic outcomes were used at the interim analysis for 18/19 of trials with results. The reporting of health economic results was suboptimal for the (17/19) trials with published results.

CONCLUSIONS

Appropriate consideration is rarely given to the health economic analysis of adaptive designs. Opportunities to use health economic outcomes in the design and analysis of adaptive trials are being missed. Further work is needed to establish whether adaptive designs and health economic analyses can be used together to increase the efficiency of health technology assessments without compromising accuracy.

Value of Information Choices that Influence Estimates: A Systematic Review of Prevailing Considerations

Background. Although value of information (VOI) analyses are increasingly advocated and used for research prioritization and reimbursement decisions, the interpretation and usefulness of VOI outcomes depend critically on the underlying choices and assumptions used in the analysis. In this article, we present a structured overview of all items reported in literature to potentially influence VOI outcomes. Use of this overview increases awareness and transparency of choices and assumptions underpinning VOI outcomes. Methods. A systematic literature review was performed to identify aspects of VOI analyses that were found to potentially influence VOI outcomes. Identified aspects were grouped to develop a structured overview. Explanations were defined for all items included in the overview. Results. We retrieved 687 unique papers, of which 71 original papers and 8 reviews were included. In the full text of these 79 papers, 16 aspects were found that may influence VOI outcomes. These aspects related to the underlying evidence (bias, synthesis, heterogeneity, correlation), uncertainty (structural, future pricing), model (relevance, approach, population), choices in VOI calculation (estimation technique, implementation level, population size, perspective), and aspects specifically for assessing the value of future study designs (reversal costs, efficient estimator). These aspects were aggregated into 7 items to provide a structured overview. Conclusion. The developed overview should increase awareness of key choices underlying VOI analysis and facilitate structured reporting of such choices and interpretation of the ensuing VOI outcomes by researchers and policy makers. Use of this overview should improve prioritization and reimbursement decisions.

PROMISE: first-trimester progesterone therapy in women with a history of unexplained recurrent miscarriages - a randomised, double-blind, placebo-controlled, international multicentre trial and economic evaluation

BACKGROUND AND OBJECTIVES

Progesterone is essential to maintain a healthy pregnancy. Guidance from the Royal College of Obstetricians and Gynaecologists and a Cochrane review called for a definitive trial to test whether or not progesterone therapy in the first trimester could reduce the risk of miscarriage in women with a history of unexplained recurrent miscarriage (RM). The PROMISE trial was conducted to answer this question. A concurrent cost-effectiveness analysis was conducted.

DESIGN AND SETTING

A randomised, double-blind, placebo-controlled, international multicentre study, with economic evaluation, conducted in hospital settings across the UK (36 sites) and in the Netherlands (nine sites).

PARTICIPANTS AND INTERVENTIONS

Women with unexplained RM (three or more first-trimester losses), aged between 18 and 39 years at randomisation, conceiving naturally and giving informed consent, received either micronised progesterone (Utrogestan(®), Besins Healthcare) at a dose of 400 mg (two vaginal capsules of 200 mg) or placebo vaginal capsules twice daily, administered vaginally from soon after a positive urinary pregnancy test (and no later than 6 weeks of gestation) until 12 completed weeks of gestation (or earlier if the pregnancy ended before 12 weeks).

MAIN OUTCOME MEASURES

Live birth beyond 24 completed weeks of gestation (primary outcome), clinical pregnancy at 6-8 weeks, ongoing pregnancy at 12 weeks, miscarriage, gestation at delivery, neonatal survival at 28 days of life, congenital abnormalities and resource use.

METHODS

Participants were randomised after confirmation of pregnancy. Randomisation was performed online via a secure internet facility. Data were collected on four occasions of outcome assessment after randomisation, up to 28 days after birth.

RESULTS

A total of 1568 participants were screened for eligibility. Of the 836 women randomised between 2010 and 2013, 404 received progesterone and 432 received placebo. The baseline data (age, body mass index, maternal ethnicity, smoking status and parity) of the participants were comparable in the two arms of the trial. The follow-up rate to primary outcome was 826 out of 836 (98.8%). The live birth rate in the progesterone group was 65.8% (262/398) and in the placebo group it was 63.3% (271/428), giving a relative risk of 1.04 (95% confidence interval 0.94 to 1.15; p = 0.45). There was no evidence of a significant difference between the groups for any of the secondary outcomes. Economic analysis suggested a favourable incremental cost-effectiveness ratio for decision-making but wide confidence intervals indicated a high level of uncertainty in the health benefits. Additional sensitivity analysis suggested the probability that progesterone would fall within the National Institute for Health and Care Excellence's threshold of £20,000-30,000 per quality-adjusted life-year as between 0.7145 and 0.7341.

CONCLUSIONS

There is no evidence that first-trimester progesterone therapy improves outcomes in women with a history of unexplained RM.

LIMITATIONS

This study did not explore the effect of treatment with other progesterone preparations or treatment during the luteal phase of the menstrual cycle.

FUTURE WORK

Future research could explore the efficacy of progesterone supplementation administered during the luteal phase of the menstrual cycle in women attempting natural conception despite a history of RM.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN92644181; EudraCT 2009-011208-42; Research Ethics Committee 09/H1208/44.

FUNDING

This project was funded by the National Institute for Health Research (NIHR) Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 20, No. 41. See the NIHR Journals Library website for further project information.

Models and applications for measuring the impact of health research: update of a systematic review for the Health Technology Assessment programme

BACKGROUND

This report reviews approaches and tools for measuring the impact of research programmes, building on, and extending, a 2007 review.

OBJECTIVES

(1) To identify the range of theoretical models and empirical approaches for measuring the impact of health research programmes; (2) to develop a taxonomy of models and approaches; (3) to summarise the evidence on the application and use of these models; and (4) to evaluate the different options for the Health Technology Assessment (HTA) programme.

DATA SOURCES

We searched databases including Ovid MEDLINE, EMBASE, Cumulative Index to Nursing and Allied Health Literature and The Cochrane Library from January 2005 to August 2014.

REVIEW METHODS

This narrative systematic literature review comprised an update, extension and analysis/discussion. We systematically searched eight databases, supplemented by personal knowledge, in August 2014 through to March 2015.

RESULTS

The literature on impact assessment has much expanded. The Payback Framework, with adaptations, remains the most widely used approach. It draws on different philosophical traditions, enhancing an underlying logic model with an interpretative case study element and attention to context. Besides the logic model, other ideal type approaches included constructionist, realist, critical and performative. Most models in practice drew pragmatically on elements of several ideal types. Monetisation of impact, an increasingly popular approach, shows a high return from research but relies heavily on assumptions about the extent to which health gains depend on research. Despite usually requiring systematic reviews before funding trials, the HTA programme does not routinely examine the impact of those trials on subsequent systematic reviews. The York/Patient-Centered Outcomes Research Institute and the Grading of Recommendations Assessment, Development and Evaluation toolkits provide ways of assessing such impact, but need to be evaluated. The literature, as reviewed here, provides very few instances of a randomised trial playing a major role in stopping the use of a new technology. The few trials funded by the HTA programme that may have played such a role were outliers.

DISCUSSION

The findings of this review support the continued use of the Payback Framework by the HTA programme. Changes in the structure of the NHS, the development of NHS England and changes in the National Institute for Health and Care Excellence's remit pose new challenges for identifying and meeting current and future research needs. Future assessments of the impact of the HTA programme will have to take account of wider changes, especially as the Research Excellence Framework (REF), which assesses the quality of universities' research, seems likely to continue to rely on case studies to measure impact. The HTA programme should consider how the format and selection of case studies might be improved to aid more systematic assessment. The selection of case studies, such as in the REF, but also more generally, tends to be biased towards high-impact rather than low-impact stories. Experience for other industries indicate that much can be learnt from the latter. The adoption of researchfish^® (researchfish Ltd, Cambridge, UK) by most major UK research funders has implications for future assessments of impact. Although the routine capture of indexed research publications has merit, the degree to which researchfish will succeed in collecting other, non-indexed outputs and activities remains to be established.

LIMITATIONS

There were limitations in how far we could address challenges that faced us as we extended the focus beyond that of the 2007 review, and well beyond a narrow focus just on the HTA programme.

CONCLUSIONS

Research funders can benefit from continuing to monitor and evaluate the impacts of the studies they fund. They should also review the contribution of case studies and expand work on linking trials to meta-analyses and to guidelines.

FUNDING

The National Institute for Health Research HTA programme.

Behavioural Activation Therapy for Depression after Stroke (BEADS): a study protocol for a feasibility randomised controlled pilot trial of a psychological intervention for post-stroke depression

BACKGROUND

There is currently insufficient evidence for the clinical and cost-effectiveness of psychological therapies for treating post-stroke depression.

METHODS/DESIGN

BEADS is a parallel group feasibility multicentre randomised controlled trial with nested qualitative research and economic evaluation. The aim is to evaluate the feasibility of undertaking a full trial comparing behavioural activation (BA) to usual stroke care for 4 months for patients with post-stroke depression. We aim to recruit 72 patients with post-stroke depression over 12 months at three centres, with patients identified from the National Health Service (NHS) community and acute services and from the voluntary sector. They will be randomly allocated to receive behavioural activation in addition to usual care or usual care alone. Outcomes will be measured at 6 months after randomisation for both participants and their carers, to determine their effectiveness. The primary clinical outcome measure for the full trial will be the Patient Health Questionnaire-9 (PHQ-9). Rates of consent, recruitment and follow-up by centre and randomised group will be reported. The acceptability of the intervention to patients, their carers and therapists will also be assessed using qualitative interviews. The economic evaluation will be undertaken from the National Health Service and personal social service perspective, with a supplementary analysis from the societal perspective. A value of information analysis will be completed to identify the areas in which future research will be most valuable.

DISCUSSION

The feasibility outcomes from this trial will provide the data needed to inform the design of a definitive multicentre randomised controlled trial evaluating the clinical and cost-effectiveness of behavioural activation for treating post-stroke depression.

TRIAL REGISTRATION

Current controlled trials ISRCTN12715175.

Value of information analysis for interventional and counterfactual Bayesian networks in forensic medical sciences

OBJECTIVES

Inspired by real-world examples from the forensic medical sciences domain, we seek to determine whether a decision about an interventional action could be subject to amendments on the basis of some incomplete information within the model, and whether it would be worthwhile for the decision maker to seek further information prior to suggesting a decision.

METHOD

The method is based on the underlying principle of Value of Information to enhance decision analysis in interventional and counterfactual Bayesian networks.

RESULTS

The method is applied to two real-world Bayesian network models (previously developed for decision support in forensic medical sciences) to examine the average gain in terms of both Value of Information (average relative gain ranging from 11.45% and 59.91%) and decision making (potential amendments in decision making ranging from 0% to 86.8%).

CONCLUSIONS

We have shown how the method becomes useful for decision makers, not only when decision making is subject to amendments on the basis of some unknown risk factors, but also when it is not. Knowing that a decision outcome is independent of one or more unknown risk factors saves us from the trouble of seeking information about the particular set of risk factors. Further, we have also extended the assessment of this implication to the counterfactual case and demonstrated how answers about interventional actions are expected to change when some unknown factors become known, and how useful this becomes in forensic medical science.

Cost-effectiveness analysis alongside clinical trials II-An ISPOR Good Research Practices Task Force report

Clinical trials evaluating medicines, medical devices, and procedures now commonly assess the economic value of these interventions. The growing number of prospective clinical/economic trials reflects both widespread interest in economic information for new technologies and the regulatory and reimbursement requirements of many countries that now consider evidence of economic value along with clinical efficacy. As decision makers increasingly demand evidence of economic value for health care interventions, conducting high-quality economic analyses alongside clinical studies is desirable because they broaden the scope of information available on a particular intervention, and can efficiently provide timely information with high internal and, when designed and analyzed properly, reasonable external validity. In 2005, ISPOR published the Good Research Practices for Cost-Effectiveness Analysis Alongside Clinical Trials: The ISPOR RCT-CEA Task Force report. ISPOR initiated an update of the report in 2014 to include the methodological developments over the last 9 years. This report provides updated recommendations reflecting advances in several areas related to trial design, selecting data elements, database design and management, analysis, and reporting of results. Task force members note that trials should be designed to evaluate effectiveness (rather than efficacy) when possible, should include clinical outcome measures, and should obtain health resource use and health state utilities directly from study subjects. Collection of economic data should be fully integrated into the study. An incremental analysis should be conducted with an intention-to-treat approach, complemented by relevant subgroup analyses. Uncertainty should be characterized. Articles should adhere to established standards for reporting results of cost-effectiveness analyses. Economic studies alongside trials are complementary to other evaluations (e.g., modeling studies) as information for decision makers who consider evidence of economic value along with clinical efficacy when making resource allocation decisions.

Clinical and cost effectiveness of computer treatment for aphasia post stroke (Big CACTUS): study protocol for a randomised controlled trial

BACKGROUND

Aphasia affects the ability to speak, comprehend spoken language, read and write. One third of stroke survivors experience aphasia. Evidence suggests that aphasia can continue to improve after the first few months with intensive speech and language therapy, which is frequently beyond what resources allow. The development of computer software for language practice provides an opportunity for self-managed therapy. This pragmatic randomised controlled trial will investigate the clinical and cost effectiveness of a computerised approach to long-term aphasia therapy post stroke.

METHODS/DESIGN

A total of 285 adults with aphasia at least four months post stroke will be randomly allocated to either usual care, computerised intervention in addition to usual care or attention and activity control in addition to usual care. Those in the intervention group will receive six months of self-managed word finding practice on their home computer with monthly face-to-face support from a volunteer/assistant. Those in the attention control group will receive puzzle activities, supplemented by monthly telephone calls. Study delivery will be coordinated by 20 speech and language therapy departments across the United Kingdom. Outcome measures will be made at baseline, six, nine and 12 months after randomisation by blinded speech and language therapist assessors. Primary outcomes are the change in number of words (of personal relevance) named correctly at six months and improvement in functional conversation. Primary outcomes will be analysed using a Hochberg testing procedure. Significance will be declared if differences in both word retrieval and functional conversation at six months are significant at the 5% level, or if either comparison is significant at 2.5%. A cost utility analysis will be undertaken from the NHS and personal social service perspective. Differences between costs and quality-adjusted life years in the three groups will be described and the incremental cost effectiveness ratio will be calculated. Treatment fidelity will be monitored.

DISCUSSION

This is the first fully powered trial of the clinical and cost effectiveness of computerised aphasia therapy. Specific challenges in designing the protocol are considered.

TRIAL REGISTRATION

Registered with Current Controlled Trials ISRCTN68798818 on 18 February 2014.

Evaluation of patient registries supporting reimbursement decisions: the case of oxaliplatin for treatment of stage III colon cancer

BACKGROUND

Access with evidence development has been established for expensive intramural drugs in The Netherlands. The procedure involves a 4-year period of conditional reimbursement. During this period, additional evidence has to be gathered usually through a patient registry. Given the costs and time involved in gathering the data, it is important to carefully evaluate the registry.

OBJECTIVES

This study aimed to develop a model for the regular evaluation of patient registries during an access with evidence development process and find the optimal length of the registry period.

METHODS

We used data from a recent registry in The Netherlands on oxaliplatin as a treatment option for stage III colon cancer. We added simulated follow-up data to the empirical data available and applied value of information analysis to balance the gains of extending the period and amount of data gathering against the costs of registering patients.

RESULTS

We show that given the assumptions on cohort size, follow-up time, and purpose of the registry, the current (partly simulated) registry was not very efficient. Notably, the observation period could have been stopped to make a definite reimbursement decision after a maximum of 2 years rather than the fixed 4-year period.

CONCLUSIONS

Patient registries may be an efficient way to gather data on new medical treatments, but they need to be carefully designed and evaluated, in particular regarding their follow-up time. For each purpose, data gathering can be tailored to make sure decisions are taken at the moment that sufficient data are available.

Value-engineered translation for regenerative medicine: meeting the needs of health systems

Despite high expectations of economic returns, large investments in regenerative medicine technology have yet to materialize, partly due to a lack of proven business and investment models, regulatory hurdles, and a greater focus on cost-effectiveness for reimbursement decisions by payors. Adoption of new economic modeling methods will better link investment decisions to value-based criteria of health systems.

COST-UTILITY OF SELF-MANAGED COMPUTER THERAPY FOR PEOPLE WITH APHASIA

Objectives: The aim of this study was to examine the potential cost-effectiveness of self-managed computer therapy for people with long-standing aphasia post stroke and to estimate the value of further research.

Methods: The incremental cost-effectiveness ratio of computer therapy in addition to usual stimulation compared with usual stimulation alone was considered in people with long-standing aphasia using data from the CACTUS trial. A model-based approach was taken. Where possible the input parameters required for the model were obtained from the CACTUS trial data, a United Kingdom-based pilot randomized controlled trial that recruited thirty-four people with aphasia and randomized them to computer treatment or usual care. Cost-effectiveness was described using an incremental cost-effectiveness ratio (ICER) together with cost-effectiveness acceptability curves. A value of information analysis was undertaken to inform future research priorities.

Results: The intervention had an ICER of £3,058 compared with usual care. The likelihood of the intervention being cost-effective was 75.8 percent at a cost-effectiveness threshold of £20,000 per QALY gained. The expected value of perfect information was £37 million.

Conclusions: Our results suggest that computer therapy for people with long-standing aphasia is likely to represent a cost-effective use of resources. However, our analysis is exploratory given the small size of the trial it is based upon and therefore our results are uncertain. Further research would be of high value, particularly with respect to the quality of life gain achieved by people who respond well to therapy.

Value of information analysis in oncology: the value of evidence and evidence of value

PURPOSE

Value of information (VOI) analysis is a novel systematic approach for assessing whether there is sufficient evidence to support regulatory approval of new technologies, estimating the value of additional research, informing trial design, and setting research priorities. This article reviews the use of VOI methods in oncology and identifies the potential applications of VOI in this field.

METHODS

A systematic literature search was undertaken to identify studies explicitly reporting VOI analyses for interventions directed at cancer management. Articles published from 2000 onward addressing prevention, screening, diagnosis, treatment, or symptom management in oncology were selected.

RESULTS

A total of 35 articles were included in the review; most were published after 2006. The main cancers addressed were breast (n = 10; 29%), prostate (n = 5; 14%), lung (n = 5; 14%), and colorectal (n = 3; 9%). The VOI analyses were of an applied nature in 31 studies (89%). In the applied studies, VOI was used to characterize decision uncertainty in all studies and to inform future research focus in 16 (52%). Additionally, one article (3%) addressed the value of optimal trial design, and one article (3%) reported the use of VOI methods to prioritize research.

CONCLUSION

The application of VOI analysis in oncology is growing but remains limited. Benefits in oncology research and practice will potentially be optimized with an increase in the application of VOI methods to inform decision making, optimal trial design, and research prioritization in this field.

Expected value of sample information for multi-arm cluster randomized trials with binary outcomes

Expected value of sample information (EVSI) measures the anticipated net benefit gained from conducting new research with a specific design to add to the evidence on which reimbursement decisions are made. Cluster randomized trials raise specific issues for EVSI calculations because 1) a hierarchical model is necessary to account for between-cluster variability when incorporating new evidence and 2) heterogeneity between clusters needs to be carefully characterized in the cost-effectiveness analysis model. Multi-arm trials provide parameter estimates that are correlated, which needs to be accounted for in EVSI calculations. Furthermore, EVSI is computationally intensive when the net benefit function is nonlinear, due to the need for an inner-simulation step. We develop a method for the computation of EVSI that avoids the inner simulation step for cluster randomized multi-arm trials with a binary outcome, where the net benefit function is linear in the probability of an event but nonlinear in the log-odds ratio parameters. We motivate and illustrate the method with an example of a cluster randomized 2 × 2 factorial trial for interventions to increase attendance at breast screening in the UK, using a previously reported cost-effectiveness model. We highlight assumptions made in our approach, extensions to individually randomized trials and inclusion of covariates, and areas for further developments. We discuss computation time, the research-design space, and the ethical implications of an EVSI approach. We suggest that EVSI is a practical and appropriate tool for the design of cluster randomized trials.

Using value of information to guide evaluation of decision supports for differential diagnosis: is it time for a new look?

BACKGROUND

Decision support systems for differential diagnosis have traditionally been evaluated on the basis of criteria how sensitively and specifically they are able to identify the correct diagnosis established by expert clinicians.

DISCUSSION

This article questions whether evaluation criteria pertaining to identifying the correct diagnosis are most appropriate or useful. Instead it advocates evaluation of decision support systems for differential diagnosis based on the criterion of maximizing value of information.

SUMMARY

This approach quantitatively and systematically integrates several important clinical management priorities, including avoiding serious diagnostic errors of omission and avoiding harmful or expensive tests.

A systematic and critical review of the evolving methods and applications of value of information in academia and practice

OBJECTIVE

This article provides a systematic and critical review of the evolving methods and applications of value of information (VOI) in academia and practice and discusses where future research needs to be directed.

METHODS

Published VOI studies were identified by conducting a computerized search on Scopus and ISI Web of Science from 1980 until December 2011 using pre-specified search terms. Only full-text papers that outlined and discussed VOI methods for medical decision making, and studies that applied VOI and explicitly discussed the results with a view to informing healthcare decision makers, were included. The included papers were divided into methodological and applied papers, based on the aim of the study.

RESULTS

A total of 118 papers were included of which 50 % (n = 59) are methodological. A rapidly accumulating literature base on VOI from 1999 onwards for methodological papers and from 2005 onwards for applied papers is observed. Expected value of sample information (EVSI) is the preferred method of VOI to inform decision making regarding specific future studies, but real-life applications of EVSI remain scarce. Methodological challenges to VOI are numerous and include the high computational demands, dealing with non-linear models and interdependency between parameters, estimations of effective time horizons and patient populations, and structural uncertainties.

CONCLUSION

VOI analysis receives increasing attention in both the methodological and the applied literature bases, but challenges to applying VOI in real-life decision making remain. For many technical and methodological challenges to VOI analytic solutions have been proposed in the literature, including leaner methods for VOI. Further research should also focus on the needs of decision makers regarding VOI.

Computer modeling informs study design: vaginal estrogen to prevent mesh erosion after different routes of prolapse surgery

INTRODUCTION AND HYPOTHESIS

Many clinicians use perioperative vaginal estrogen therapy (estradiol, E(2)) to diminish the risk of mesh erosion after prolapse surgery, though supporting evidence is limited. We assessed the feasibility of a factorial randomized trial comparing mesh erosion rates after vaginal mesh prolapse surgery (VM) versus minimally invasive sacral colpopexy (MISC), with or without adjunct vaginal estrogen therapy.

METHODS

A Markov state transition model simulated the probability of 2-year outcomes of visceral injury, mesh erosion, and reoperation after four possible prolapse therapies: VM or MISC, each with or without estrogen therapy (E(2)). We used pooled estimates from a systematic review to generate probability distributions for the following outcomes after each procedure: visceral injury, postoperative mesh erosion, and reoperation for either recurrent prolapse or mesh erosion. Assuming different assumptions for E(2) efficacies (50 and 75 % reduction in erosion rates), Monte Carlo simulations estimated outcomes rates, which were then used to generate sample size estimates for a four-arm factorial trial.

RESULTS

While E(2) reduced the risk of mesh erosion for both VM and MISC, absolute reduction was small. Assuming 75 % efficacy, E(2) decreased the risk of mesh erosion for VM from 7.8 to 2.0 % and for MISC from 2.0 to 0.5 %. Total sample sizes ranged from 448 to 1,620, depending on power and E(2) efficacy.

CONCLUSIONS

The required sample size for a trial to determine which therapy results in the lowest erosion rates would be prohibitively large. Because this remains an important clinical issue, further study design strategies could include composite outcomes, cost-effectiveness, or value of information analysis.

Expected net present value of sample information: from burden to investment

The Expected Value of Information Framework has been proposed as a method for identifying when health care technologies should be immediately reimbursed and when any reimbursement should be withheld while awaiting more evidence. This framework assesses the value of obtaining additional evidence to inform a current reimbursement decision. This represents the burden of not having the additional evidence at the time of the decision. However, when deciding whether to reimburse now or await more evidence, decision makers need to know the value of investing in more research to inform a future decision. Assessing this value requires consideration of research costs, research time, and what happens to patients while the research is undertaken and after completion. The investigators describe a development of the calculation of the expected value of sample information that assesses the value of investing in further research, including an only-in-research strategy and an only-with-research strategy.

The effect of aspirin and low-molecular-weight heparin on venous thromboembolism after hip replacement: a non-randomised comparison from information in the National Joint Registry

We compared thromboembolic events, major haemorrhage and death after total hip replacement in patients receiving either aspirin or low-molecular-weight heparin (LMWH). We analysed data from the National Joint Registry for England and Wales linked to an administrative database of hospital admissions in the English National Health Service. A total of 108,584 patients operated on between April 2003 and September 2008 were included and followed up for 90 days. Multivariable risk modelling and propensity score matching were used to estimate odds ratios (OR) adjusted for baseline risk factors. An OR < 1 indicates that rates are lower with LMWH than with aspirin. In all, 21.1% of patients were prescribed aspirin and 78.9% LMWH. Without adjustment, we found no statistically significant differences. The rate of pulmonary embolism was 0.68% in both groups and 90-day mortality was 0.65% with aspirin and 0.61% with LMWH (OR 0.93; 95% CI 0.77 to 1.11). With risk adjustment, the difference in mortality increased (OR 0.84; 95% CI 0.69 to 1.01). With propensity score matching the mortality difference increased even further to 0.65% with aspirin and 0.51% with LMWH (OR 0.77; 95% CI 0.61 to 0.98). These results should be considered when the conflicting recommendations of existing guidelines for thromboprophylaxis after hip replacement are being addressed.

A CTSA agenda to advance methods for comparative effectiveness research

Clinical research needs to be more useful to patients, clinicians, and other decision makers. To meet this need, more research should focus on patient-centered outcomes, compare viable alternatives, and be responsive to individual patients' preferences, needs, pathobiology, settings, and values. These features, which make comparative effectiveness research (CER) fundamentally patient-centered, challenge researchers to adopt or develop methods that improve the timeliness, relevance, and practical application of clinical studies. In this paper, we describe 10 priority areas that address 3 critical needs for research on patient-centered outcomes (PCOR): (1) developing and testing trustworthy methods to identify and prioritize important questions for research; (2) improving the design, conduct, and analysis of clinical research studies; and (3) linking the process and outcomes of actual practice to priorities for research on patient-centered outcomes. We argue that the National Institutes of Health, through its clinical and translational research program, should accelerate the development and refinement of methods for CER by linking a program of methods research to the broader portfolio of large, prospective clinical and health system studies it supports. Insights generated by this work should be of enormous value to PCORI and to the broad range of organizations that will be funding and implementing CER.

Addressing Adoption and Research Design Decisions Simultaneously

Methods to estimate the cost-effectiveness of technologies are well developed with increasing experience of their application to inform adoption decisions in a timely way. However, the experience of using similarly explicit methods to inform the associated research decisions is less well developed despite appropriate methods being available with an increasing number of applications in health. The authors demonstrate that evaluation of both adoption and research decisions is feasible within typical time and resource constraints relevant to policy decisions, even in situations in which data are sparse and formal elicitation is required. In addition to demonstrating the application of expected value of sample information (EVSI) in these circumstances, the authors examine and carefully distinguish the impact that the research decision is expected to have on patients while enrolled in the trial, those not enrolled, and once the trial reports. In doing so, the authors are able to account for the range of opportunity cost associated with research and evaluate a number of research designs including length of follow-up and sample size. The authors also explore the implications for research design of conducting research while the technology is approved for widespread use and whether approval should be withheld until research reports. In doing so, the authors highlight the impact of irrecoverable opportunity costs when the initial costs of a technology are compensated only by later gains in health outcome.

Value of information in the osteoarthritis setting: cost effectiveness of COX-2 selective inhibitors, traditional NSAIDs and proton pump inhibitors

BACKGROUND

Recent National Institute for Health and Clinical Excellence (NICE) guidance recommended that when traditional NSAIDs or cyclo-oxygenase (COX)-2 selective inhibitors are used by people with osteoarthritis (OA), they should be prescribed along with a proton pump inhibitor (PPI). However, specific recommendations about the type of NSAID or COX-2 could not be made due to high levels of uncertainty in the economic evaluation.

OBJECTIVE

To investigate the value of obtaining further evidence to inform the economic evaluation of NSAIDs, COX-2s and PPIs for people with OA.

METHODS

An economic evaluation with an expected value of perfect information (EVPI) analysis was conducted, using a Markov model with data identified from a systematic review. The base-case model used adverse event data from the three largest randomized trials of COX-2 inhibitors, and we repeated the analysis using observational adverse event data. The model was run for a hypothetical population of people with OA, and subgroup analyses were conducted for people with raised gastrointestinal (GI) and cardiovascular (CV) risk. The EVPI was based upon the OA population in England - approximately 2.8 million people. Of these, 50% were assumed to use NSAIDs or COX-2 selective inhibitors for 3 months per year and 56% of these were assumed to be patients with raised GI and CV risk.

RESULTS

The value of further information for this decision problem was very high. Population-level EVPI was £85.1 million in the low-risk group and £179.5 million in the high-risk group (2007-8 values). Expected value of partial perfect information (EVPPI) analysis showed that the groups of parameters for which further evidence was likely to be of most value were CV adverse event risks and all adverse event rates associated with the specific drugs celecoxib and ibuprofen. The value of perfect information remained high even when observational adverse event data were used.

CONCLUSIONS

There is a very high value associated with obtaining further information on uncertain parameters for the economic evaluation of NSAIDs, COX-2 selective inhibitors and PPIs for people with OA. Obtaining further randomized or observational information on CV risks is likely to be particularly cost effective.