Treatment success in pragmatic randomised controlled trials: a review of trials funded by the UK Health Technology Assessment programme

Constructing Relative Effect Priors for Research Prioritization and Trial Design: A Meta-epidemiological Analysis

BACKGROUND

Bayesian methods have potential for efficient design of randomized clinical trials (RCTs) by incorporating existing evidence. Furthermore, value of information (VOI) methods estimate the value of reducing decision uncertainty, aiding transparent research prioritization. These methods require a prior distribution describing current uncertainty in key parameters, such as relative treatment effect (RTE). However, at the time of designing and commissioning research, there may be no data to base the prior on. The aim of this article is to present methods to construct priors for RTEs based on a collection of previous RCTs.

METHODS

We developed 2 Bayesian hierarchical models that captured variability in RTE between studies within disease area accounting for study characteristics. We illustrate the methods using a data set of 743 published RCTs across 9 disease areas to obtain predictive distributions for RTEs for a range of disease areas. We illustrate how the priors from such an analysis can be used in a VOI analysis for an RCT in bladder cancer and compare the results with those using an uninformative prior.

RESULTS

For most disease areas, the predicted RTE favored new interventions over comparators. The predicted effects and uncertainty differed across the 9 disease areas. VOI analysis showed that the expected value of research is much lower with our empirically derived prior compared with an uninformative prior.

CONCLUSIONS

This study demonstrates a novel approach to generating informative priors that can be used to aid research prioritization and trial design. The methods can also be used to combine RCT evidence with expert opinion. Further work is needed to create a rich database of RCT evidence that can be used to form off-the-shelf priors.

HIGHLIGHTS

Bayesian methods have potential to aid the efficient design of randomized clinical trials (RCTs) by incorporating existing evidence. Value-of-information (VOI) methods can be used to aid research prioritization by calculating the value of current decision uncertainty.These methods require a distribution describing current uncertainty in key parameters, that is, "prior distributions."This article demonstrates a methodology to estimate prior distributions for relative treatment effects (odds and hazard ratios) estimated from a collection of previous RCTs.These results may be combined with expert elicitation to facilitate 1) value-of-information methods to prioritize research or 2) Bayesian methods for research design.

'Optimism bias' in contemporary national clinical trial network phase III trials: are we improving?

Background

Previous studies have found that overestimating treatment effects (i.e. 'optimism bias') leads to underpowered clinical trials. The prevalence of 'optimism bias' in contemporary National Clinical Trials Network (NCTN) cancer clinical trials is unknown.

Methods

We conducted a systematic review of NCTN phase III randomized trials published from January 2007 to January 2017. We compared the hypothesized versus observed treatment effects in each trial, and examined whether trial-related factors were correlated with the study results. We also reviewed the methods of each protocol to assess whether a rationale for the hypothesized effect size was provided.

Results

We identified 161 clinical trials, of which 130 were eligible for analysis. Original protocols could not be located for 8 trials (5.0%). Twenty-eight trials (21.5%) observed a statistically significant difference in the primary end point favoring the experimental treatment. The median ratio of observed-to-expected hazard ratios among trials that observed a statistically significant effect on the primary end point was 1.07 (range: 0.33-1.28) versus 1.32 (range: 0.86-2.02) for trials that did not, compared with 1.34 and 1.86, respectively, for National Cancer Institute (NCI) trials published between 1955 and 2006. An effect size at least as large as the one projected in the protocol trials was observed in 9.8% of trials, compared with 17% of NCI trials published from 1955 to 2006. Most trials (64.6%) provided no rationale to support the magnitude of the proposed treatment effect in the protocol.

Conclusions

Despite a reduction in 'optimism bias' compared with previous eras, most contemporary NCTN phase III trials failed to establish statistically significant benefits of new cancer therapies. Better rationalization of proposed effect sizes in research protocols is needed.

Intervention development and treatment success in UK health technology assessment funded trials of physical rehabilitation: a mixed methods analysis

OBJECTIVES

Physical rehabilitation is a complex process, and trials of rehabilitation interventions are increasing in number but often report null results. This study aimed to establish treatment success rates in physical rehabilitation trials funded by the National Institute of Health Research Health Technology Assessment (NIHR HTA) programme and examine any relationship between treatment success and the quality of intervention development work undertaken.

DESIGN

This is a mixed methods study.

SETTING

This study was conducted in the UK.

METHODS

The NIHR HTA portfolio was searched for all completed definitive randomised controlled trials of physical rehabilitation interventions from inception to July 2016. Treatment success was categorised according to criteria developed by Djulbegovic and colleagues. Detailed textual data regarding any intervention development work were extracted from trial reports and supporting publications and informed the development of quality ratings. Mixed methods integrative analysis was undertaken to explore the relationship between quantitative and qualitative data using joint displays.

RESULTS

Fifteen trials were included in the review. Five reported a definitive finding, four of which were in favour of the 'new' intervention. Eight trials reported a true negative (no difference) outcome. Integrative analysis indicated those with lower quality intervention development work were less likely to report treatment success.

CONCLUSIONS

Despite much effort and funding, most physical rehabilitation trials report equivocal findings. Greater focus on high quality intervention development may reduce the likelihood of a null result in the definitive trial, alongside high quality trial methods and conduct.

The Second Triennial Systematic Literature Review of European Nursing Research: Impact on Patient Outcomes and Implications for Evidence-Based Practice

BACKGROUND

European research in nursing has been criticized as overwhelmingly descriptive, wasteful and with little relevance to clinical practice. This second triennial review follows our previous review of articles published in 2010, to determine whether the situation has changed.

OBJECTIVE

To identify, appraise, and synthesize reports of European nursing research published during 2013 in the top 20 nursing research journals.

METHODS

Systematic review with descriptive results synthesis.

RESULTS

We identified 2,220 reports, of which 254, from 19 European countries, were eligible for analysis; 215 (84.7%) were primary research, 36 (14.2%) secondary research, and three (1.2%) mixed primary and secondary. Forty-eight (18.9%) of studies were experimental: 24 (9.4%) randomized controlled trials, 11 (4.3%) experiments without randomization, and 13 (5.1%) experiments without control group. A total of 106 (41.7%) articles were observational: 85 (33.5%) qualitative research. The majority (158; 62.2%) were from outpatient and secondary care hospital settings. One hundred and sixty-five (65.0%) articles reported nursing intervention studies: 77 (30.3%) independent interventions, 77 (30.3%) interdependent, and 11 (4.3%) dependent. This represents a slight increase in experimental studies compared with our previous review (18.9% vs. 11.7%). The quality of reporting remained very poor.

LINKING EVIDENCE TO ACTION

European research in nursing remains overwhelmingly descriptive. We call on nursing researchers globally to raise the level of evidence and, therefore, the quality of care and patient outcomes. We urge them to replicate our study in their regions, diagnose reasons for the lack of appropriate research, identify solutions, and implement a deliberate, targeted, and systematic global effort to increase the number of experimental, high quality, and relevant studies into nursing interventions. We also call on journal editors to mandate an improvement in the standards of research reporting in nursing journals.

Increasing value and reducing waste by optimizing the development of complex interventions: Enriching the development phase of the Medical Research Council (MRC) Framework

BACKGROUND

In recent years there has been much emphasis on 'research waste' caused by poor question selection, insufficient attention to previous research results, and avoidable weakness in research design, conduct and analysis. Little attention has been paid to the effect of inadequate development of interventions before proceeding to a full clinical trial.

OBJECTIVE

We therefore propose to enrich the development phase of the MRC Framework by adding crucial elements to improve the likelihood of success and enhance the fit with clinical practice METHODS: Based on existing intervention development guidance and synthesis, a comprehensive iterative intervention development approach is proposed. Examples from published reports are presented to illustrate the methodology that can be applied within each element to enhance the intervention design.

RESULTS

A comprehensive iterative approach is presented by combining the elements of the MRC Framework development phase with essential elements from existing guidance including: problem identification, the systematic identification of evidence, identification or development of theory, determination of needs, the examination of current practice and context, modelling the process and expected outcomes leading to final element: the intervention design. All elements are drawn from existing models to provide intervention developers with a greater chance of producing an intervention that is well adopted, effective and fitted to the context.

CONCLUSION

This comprehensive approach of developing interventions will strengthen the internal and external validity, minimize research waste and add value to health care research. In complex interventions in health care research, flaws in the development process immediately impact the chances of success. Knowledge regarding the causal mechanisms and interactions within the intended clinical context is needed to develop interventions that fit daily practice and are beneficial for the end-user.

Clinical trial metadata: defining and extracting metadata on the design, conduct, results and costs of 125 randomised clinical trials funded by the National Institute for Health Research Health Technology Assessment programme

BACKGROUND

By 2011, the Health Technology Assessment (HTA) programme had published the results of over 100 trials with another 220 in progress. The aim of the project was to develop and pilot 'metadata' on clinical trials funded by the HTA programme.

OBJECTIVES

The aim of the project was to develop and pilot questions describing clinical trials funded by the HTA programme in terms of it meeting the needs of the NHS with scientifically robust studies. The objectives were to develop relevant classification systems and definitions for use in answering relevant questions and to assess their utility.

DATA SOURCES

Published monographs and internal HTA documents.

REVIEW METHODS

A database was developed, 'populated' using retrospective data and used to answer questions under six prespecified themes. Questions were screened for feasibility in terms of data availability and/or ease of extraction. Answers were assessed by the authors in terms of completeness, success of the classification system used and resources required. Each question was scored to be retained, amended or dropped.

RESULTS

One hundred and twenty-five randomised trials were included in the database from 109 monographs. Neither the International Standard Randomised Controlled Trial Number nor the term 'randomised trial' in the title proved a reliable way of identifying randomised trials. Only limited data were available on how the trials aimed to meet the needs of the NHS. Most trials were shown to follow their protocols but updates were often necessary as hardly any trials recruited as planned. Details were often lacking on planned statistical analyses, but we did not have access to the relevant statistical plans. Almost all the trials reported on cost-effectiveness, often in terms of both the primary outcome and quality-adjusted life-years. The cost of trials was shown to depend on the number of centres and the duration of the trial. Of the 78 questions explored, 61 were well answered, 33 fully with 28 requiring amendment were the analysis updated. The other 17 could not be answered with readily available data.

LIMITATIONS

The study was limited by being confined to 125 randomised trials by one funder.

CONCLUSIONS

Metadata on randomised controlled trials can be expanded to include aspects of design, performance, results and costs. The HTA programme should continue and extend the work reported here.

FUNDING

The National Institute for Health Research HTA programme.

Veterinary homeopathy: systematic review of medical conditions studied by randomised trials controlled by other than placebo

Background

No systematic review has previously been carried out on randomised controlled trials (RCTs) of veterinary homeopathy in which the control group was an intervention other than placebo (OTP). For eligible peer-reviewed RCTs, the objectives of this study were to assess the risk of bias (RoB) and to quantify the effect size of homeopathic intervention compared with an active comparator or with no treatment.

Methods

Our systematic review approach complied fully with the PRISMA 2009 Checklist. Cochrane methods were applied to assess RoB and to derive effect size using standard meta-analysis methods. Based on a thorough and systematic literature search, the following key attributes of the published research were distinguished: individualised homeopathy (n = 1 RCT)/non-individualised homeopathy (n = 19); treatment (n = 14)/prophylaxis (n = 6); active controls (n = 18)/untreated controls (n = 2). The trials were highly diverse, representing 12 different medical conditions in 6 different species.

Results

No trial had sufficiently low RoB to be judged as reliable evidence: 16 of the 20 RCTs had high RoB; the remaining four had uncertain RoB in several domains of assessment. For three trials with uncertain RoB and without overt vested interest, it was inconclusive whether homeopathy combined with conventional intervention was more or was less effective than conventional intervention alone for modulation of immune response in calves, or in the prophylaxis of cattle tick or of diarrhoea in piglets.

Conclusion

Due to the poor reliability of their data, OTP-controlled trials do not currently provide useful insight into the effectiveness of homeopathy in animals.

Electronic supplementary material

The online version of this article (doi:10.1186/s12917-015-0542-2) contains supplementary material, which is available to authorized users.

Control treatments in biologics trials of rheumatoid arthritis were often not deemed acceptable in the context of care

OBJECTIVES

Control treatments in randomized controlled trials (RCTs) should not deliberately disadvantage patients. The objectives of the study were to compare (1) willingness to include vs. (2) willingness to prescribe control treatment among physicians randomized to assess, respectively, either (1) enrollment in a trial or (2) appropriateness of control treatment in a care context for the same fictional patient.

STUDY DESIGN AND SETTING

Physicians were authors of articles about rheumatoid arthritis (RA), involved in RA patient care, and used to enrolling patients in trials. The outcomes were willingness to give control treatment: trial enrollment or control-treatment appropriateness in care context. We derived three case vignettes of fictional standard eligible patients for each of 30 RCTs assessing biologics in RA. Physicians were randomly allocated to the "trial" or "care" arm. For each of the 90 fictional patients, physicians assigned to the trial arm were asked if they would enroll the patient in the RCT the patient was derived from. For the same 90 fictional patients, physicians assigned to the care arm were asked if the control treatment of the RCT was appropriate in a context of usual care.

RESULTS

Of the 1,779 physicians invited to participate, 151 were randomized. Half of the fictional patients {41/90; 45% [95% confidence interval (CI): 37%, 53%]} would be enrolled in the RCT although the control-arm treatment of the RCT was not considered appropriate for them in the context of care. This rate differed by type of comparator [55% for non-head-to-head RCTs vs. 6% for head-to-head RCTs; adjusted odds ratio (aOR), 23.9 (95% CI: 5.5, 92.7)] and duration of trial control treatment [56% for ≤24 weeks and 15% for >24 weeks; aOR, 10.7 (95% CI: 2.8, 63.9)] but not patient RA activity [aOR, 2.5 (95% CI: 1.0, 6.6)]. The limitation of this study was that physicians gave their opinion on fictional patients with only RA.

CONCLUSIONS

Control treatments in RCTs of biologics in RA are often deemed not acceptable in the context of usual care, especially those for non-head-to-head RCTs. These findings raise ethical concerns and challenge the choice of the comparator in RCTs.

A framework to assess the value of application of formal criteria to check clinical relevance in RCTs as part of a benefit assessment strategy

Recently, the topic of assessing clinical relevance on top of statistical significance in the analysis of randomized control trials (RCTs) has got increasing attention, in particular as part of benefit assessments. Several formal criteria to serve this purpose have been published. In this paper, we present a framework to assess the value of the application of such criteria. We propose to quantify the need for the assessment of clinical relevance by the actual risk of having accepted a benefit for a treatment with an irrelevant effect in a successful RCT. We then study how this risk can be controlled by two popular criteria based on comparing the effect estimate or the lower bound of the confidence interval with a given threshold. We further propose to quantify the impact of using formal criteria by considering the expected costs when specifying error-specific costs for each of the three possible types of errors: A benefit may be accepted for a treatment, which is actually inferior, or which is not inferior, but only implies an irrelevant improvement, or a benefit may be rejected for a treatment implying a relevant improvement. This way we can demonstrate that the impact depends on parameters which are typically not explicitly defined in the frame of benefit assessments. Depending on the values of these parameters, formal checks of clinical relevance may imply better decisions on average, but they may also imply more harm than good on average.

An investigation of the impact of futility analysis in publicly funded trials

Background

Publicly funded trials regularly fail to recruit their target sample size or find a significant positive result. Adaptive clinical trials which may partly mediate against the problems are not often applied. In this paper we investigate the potential of a form of adaption in a clinical trial - a futility analysis - to see if it has potential to improve publicly funded trials.

Methods

Outcome data from trials funded by two UK bodies, the Health Technology Assessment (HTA) programme and the UK Medical Research Council (MRC), were collected. These data were then used to simulate each trial with a single futility analysis using conditional power, undertaken after 50% to 90% of the patients had been recruited. Thirty-three trials recruiting between 2002 and 2008 met the inclusion criteria. Stopping boundaries of conditional powers of 20%, 30% and 40% were considered and outcomes included the number of trials successfully stopped and number of patients saved.

Results

Inclusion of a futility analysis after 75% of the patients had been recruited would have potentially resulted in 10 trials, which went on to have negative results, correctly stopping for futility using a stopping boundary of 30%. A total of 807 patients across all the trials would potentially have been saved using these futility parameters. The proportion of studies successfully recruiting would also have increased from 45% to 64%.

Conclusions

A futility assessment has the potential to increase efficiency, save patients and decrease costs in publicly funded trials. While there are logistical issues in undertaking futility assessments we recommend that investigators should aim to include a futility analysis in their trial design wherever possible.

Health Technology Assessment in the UK

In this Review, we discuss the UK's Health Technology Assessment programme, which is 20 years old in 2013. We situate the programme in the context of the UK landscape for evidence-based medicine, including in relation to the National Institute for Health and Care Excellence and as guidance to the National Health Service. We identify features that might be of value to other health systems as they confront the challenges of rapid innovation and rising costs. We use examples of recent studies to show the strengths and weaknesses of the programme.

Treatment success in cancer: industry compared to publicly sponsored randomized controlled trials

OBJECTIVE

To assess if commercially sponsored trials are associated with higher success rates than publicly-sponsored trials.

STUDY DESIGN AND SETTINGS

We undertook a systematic review of all consecutive, published and unpublished phase III cancer randomized controlled trials (RCTs) conducted by GlaxoSmithKline (GSK) and the NCIC Clinical Trials Group (CTG). We included all phase III cancer RCTs assessing treatment superiority from 1980 to 2010. Three metrics were assessed to determine treatment successes: (1) the proportion of statistically significant trials favouring the experimental treatment, (2) the proportion of the trials in which new treatments were considered superior according to the investigators, and (3) quantitative synthesis of data for primary outcomes as defined in each trial.

RESULTS

GSK conducted 40 cancer RCTs accruing 19,889 patients and CTG conducted 77 trials enrolling 33,260 patients. 42% (99%CI 24 to 60) of the results were statistically significant favouring experimental treatments in GSK compared to 25% (99%CI 13 to 37) in the CTG cohort (RR = 1.68; p = 0.04). Investigators concluded that new treatments were superior to standard treatments in 80% of GSK compared to 44% of CTG trials (RR = 1.81; p<0.001). Meta-analysis of the primary outcome indicated larger effects in GSK trials (odds ratio = 0.61 [99%CI 0.47-0.78] compared to 0.86 [0.74-1.00]; p = 0.003). However, testing for the effect of treatment over time indicated that treatment success has become comparable in the last decade.

CONCLUSIONS

While overall industry sponsorship is associated with higher success rates than publicly-sponsored trials, the difference seems to have disappeared over time.

New treatments compared to established treatments in randomized trials

BACKGROUND

The proportion of proposed new treatments that are 'successful' is of ethical, scientific, and public importance. We investigated how often new, experimental treatments evaluated in randomized controlled trials (RCTs) are superior to established treatments.

OBJECTIVES

Our main question was: "On average how often are new treatments more effective, equally effective or less effective than established treatments?" Additionally, we wanted to explain the observed results, i.e. whether the observed distribution of outcomes is consistent with the 'uncertainty requirement' for enrollment in RCTs. We also investigated the effect of choice of comparator (active versus no treatment/placebo) on the observed results.

SEARCH METHODS

We searched the Cochrane Methodology Register (CMR) 2010, Issue 1 in The Cochrane Library (searched 31 March 2010); MEDLINE Ovid 1950 to March Week 2 2010 (searched 24 March 2010); and EMBASE Ovid 1980 to 2010 Week 11 (searched 24 March 2010).

SELECTION CRITERIA

Cohorts of studies were eligible for the analysis if they met all of the following criteria: (i) consecutive series of RCTs, (ii) registered at or before study onset, and (iii) compared new against established treatments in humans.

DATA COLLECTION AND ANALYSIS

RCTs from four cohorts of RCTs met all inclusion criteria and provided data from 743 RCTs involving 297,744 patients. All four cohorts consisted of publicly funded trials. Two cohorts involved evaluations of new treatments in cancer, one in neurological disorders, and one for mixed types of diseases. We employed kernel density estimation, meta-analysis and meta-regression to assess the probability of new treatments being superior to established treatments in their effect on primary outcomes and overall survival.

MAIN RESULTS

The distribution of effects seen was generally symmetrical in the size of difference between new versus established treatments. Meta-analytic pooling indicated that, on average, new treatments were slightly more favorable both in terms of their effect on reducing the primary outcomes (hazard ratio (HR)/odds ratio (OR) 0.91, 99% confidence interval (CI) 0.88 to 0.95) and improving overall survival (HR 0.95, 99% CI 0.92 to 0.98). No heterogeneity was observed in the analysis based on primary outcomes or overall survival (I(2) = 0%). Kernel density analysis was consistent with the meta-analysis, but showed a fairly symmetrical distribution of new versus established treatments indicating unpredictability in the results. This was consistent with the interpretation that new treatments are only slightly superior to established treatments when tested in RCTs. Additionally, meta-regression demonstrated that results have remained stable over time and that the success rate of new treatments has not changed over the last half century of clinical trials. The results were not significantly affected by the choice of comparator (active versus placebo/no therapy).

AUTHORS' CONCLUSIONS

Society can expect that slightly more than half of new experimental treatments will prove to be better than established treatments when tested in RCTs, but few will be substantially better. This is an important finding for patients (as they contemplate participation in RCTs), researchers (as they plan design of the new trials), and funders (as they assess the 'return on investment'). Although we provide the current best evidence on the question of expected 'success rate' of new versus established treatments consistent with a priori theoretical predictions reflective of 'uncertainty or equipoise hypothesis', it should be noted that our sample represents less than 1% of all available randomized trials; therefore, one should exercise the appropriate caution in interpretation of our findings. In addition, our conclusion applies to publicly funded trials only, as we did not include studies funded by commercial sponsors in our analysis.

When is it rational to participate in a clinical trial? A game theory approach incorporating trust, regret and guilt

BACKGROUND

Randomized controlled trials (RCTs) remain an indispensable form of human experimentation as a vehicle for discovery of new treatments. However, since their inception RCTs have raised ethical concerns. The ethical tension has revolved around "duties to individuals" vs. "societal value" of RCTs. By asking current patients "to sacrifice for the benefit of future patients" we risk subjugating our duties to patients' best interest to the utilitarian goal for the good of others. This tension creates a key dilemma: when is it rational, from the perspective of the trial patients and researchers (as societal representatives of future patients), to enroll in RCTs?

METHODS

We employed the trust version of the prisoner's dilemma since interaction between the patient and researcher in the setting of a clinical trial is inherently based on trust. We also took into account that the patient may have regretted his/her decision to participate in the trial, while a researcher may feel guilty because he/she abused the patient's trust.

RESULTS

We found that under typical circumstances of clinical research, most patients can be expected not to trust researchers, and most researchers can be expected to abuse the patients' trust. The most significant factor determining trust was the success of experimental or standard treatments, respectively. The more that a researcher believes the experimental treatment will be successful, the more incentive the researcher has to abuse trust. The analysis was sensitive to the assumptions about the utilities related to success and failure of therapies that are tested in RCTs. By varying all variables in the Monte Carlo analysis we found that, on average, the researcher can be expected to honor a patient's trust 41% of the time, while the patient is inclined to trust the researcher 69% of the time. Under assumptions of our model, enrollment into RCTs represents a rational strategy that can meet both patients' and researchers' interests simultaneously 19% of the time.

CONCLUSIONS

There is an inherent ethical dilemma in the conduct of RCTs. The factors that hamper full co-operation between patients and researchers in the conduct of RCTs can be best addressed by: a) having more reliable estimates on the probabilities that new vs. established treatments will be successful, b) improving transparency in the clinical trial system to ensure fulfillment of "the social contract" between patients and researchers.