Evaluation of the impact of letters to GP practices to promote asthma prescription uptake in school-age children during summer (TRAINS study): a pragmatic cluster-randomised controlled trial

BACKGROUND

Asthma exacerbations peak in school-aged children after the return to school in September. Previous studies have shown a decline in collections of asthma prescriptions during August. The PLEASANT trial demonstrated that sending a reminder letter to parents increased prescription uptake; reduced unscheduled care, and was cost saving to the health service. We aimed to assess whether informing general practitioner (GP) practices about the PLEASANT trial and its results could lead to its implementation in routine practice.

METHODS

The trial to assess implementation of new research in a primary care setting (TRAINS) was a pragmatic cluster-randomised (1:1) trial conducted in England involving GP practices contributing to the Clinical Practice Research Datalink (CPRD). The intervention was a letter informing the GP practice of the PLEASANT trial results with recommendations for implementation. GP practices in the control group continued with usual care without receiving any letters about PLEASANT trial. The intervention was distributed via CPRD by both mail and email in June 2021. The trial received both University of Sheffield Ethics approval and Independent Scientific Advisory Committee (ISAC) approval. The primary outcome was the proportion of children with asthma (aged 4-15 years) who had a prescription for a preventer between Aug 1 and Sept 30, 2021. This trial is registered with ClinicalTrials.gov, NCT05226091.

FINDINGS

A total of 1326 GP practices, including 90 583 children with asthma, were included in the study. These practices were randomly allocated to the intervention group (664 practices, 44 708 children) or the control group (662 practices, 45 875 children). In assessing the impact of the intervention on the proportion of children collecting a preventer prescription, 15 716 (35·3%) of 44 708 children from the intervention group and 16 001 (35·1%) of 45 559 children from the control group picked up a prescription. There was no statistically significant difference observed (odds ratio [OR] 1·01, 95% CI 0·97-1·05), indicating that the intervention had no effect.

INTERPRETATION

The study findings suggest that passive intervention of providing a letter to GPs did not achieve the intended outcomes. To bridge the gap between evidence and practice, alternative, more proactive strategies could be explored to address the identified issues.

FUNDING

Jazan University.

Appropriate design and reporting of superiority, equivalence and non-inferiority clinical trials incorporating a benefit-risk assessment: the BRAINS study including expert workshop

Background

Randomised controlled trials are designed to assess the superiority, equivalence or non-inferiority of a new health technology, but which trial design should be used is not always obvious in practice. In particular, when using equivalence or non-inferiority designs, multiple outcomes of interest may be important for the success of a trial, despite the fact that usually only a single primary outcome is used to design the trial. Benefit-risk methods are used in the regulatory clinical trial setting to assess multiple outcomes and consider the trade-off of the benefits against the risks, but are not regularly implemented in publicly funded trials.

Objectives

The aim of the project is to aid the design of clinical trials with multiple outcomes of interest by defining when each trial design is appropriate to use and identifying when to use benefit-risk methods to assess outcome trade-offs (qualitatively or quantitatively) in a publicly funded trial setting.

Methods

A range of methods was used to elicit expert opinion to answer the project objectives, including a web-based survey of relevant researchers, a rapid review of current literature and a 2-day consensus workshop of experts (in 2019).

Results

We created a list of 19 factors to aid researchers in selecting the most appropriate trial design, containing the following overarching sections: population, intervention, comparator, outcomes, feasibility and perspectives. Six key reasons that indicate a benefit-risk method should be considered within a trial were identified: (1) when the success of the trial depends on more than one outcome; (2) when important outcomes within the trial are in competing directions (i.e. a health technology is better for one outcome, but worse for another); (3) to allow patient preferences to be included and directly influence trial results; (4) to provide transparency on subjective recommendations from a trial; (5) to provide consistency in the approach to presenting results from a trial; and (6) to synthesise multiple outcomes into a single metric. Further information was provided to support the use of benefit-risk methods in appropriate circumstances, including the following: methods identified from the review were collated into different groupings and described to aid the selection of a method; potential implementation of methods throughout the trial process were provided and discussed (with examples); and general considerations were described for those using benefit-risk methods. Finally, a checklist of five pieces of information that should be present when reporting benefit-risk methods was defined, with two additional items specifically for reporting the results.

Conclusions

These recommendations will assist research teams in selecting which trial design to use and deciding whether or not a benefit-risk method could be included to ensure research questions are answered appropriately. Additional information is provided to support consistent use and clear reporting of benefit-risk methods in the future. The recommendations can also be used by funding committees to confirm that appropriate considerations of the trial design have been made.

Limitations

This research was limited in scope and should be considered in conjunction with other trial design methodologies to assess appropriateness. In addition, further research is needed to provide concrete information about which benefit-risk methods are best to use in publicly funded trials, along with recommendations that are specific to each method.

Study registration

The rapid review is registered as PROSPERO CRD42019144882.

Funding

Funded by the Medical Research Council UK and the National Institute for Health and Care Research as part of the Medical Research Council-National Institute for Health and Care Research Methodology Research programme.

CONSORT Harms 2022 statement, explanation, and elaboration: updated guideline for the reporting of harms in randomized trials

Randomized controlled trials remain the reference standard for healthcare research on effects of interventions, and the need to report both benefits and harms is essential. The Consolidated Standards of Reporting Trials (the main CONSORT) statement includes one item on reporting harms (i.e., all important harms or unintended effects in each group). In 2004, the CONSORT group developed the CONSORT Harms extension; however, it has not been consistently applied and needs to be updated. Here, we describe CONSORT Harms 2022, which replaces the CONSORT Harms 2004 checklist, and shows how CONSORT Harms 2022 items could be incorporated into the main CONSORT checklist. Thirteen items from the main CONSORT were modified to improve harms reporting. Three new items were added. In this article, we describe CONSORT Harms 2022 and how it was integrated into the main CONSORT checklist and elaborate on each item relevant to complete reporting of harms in randomized controlled trials. Until future work from the CONSORT group produces an updated checklist, authors, journal reviewers, and editors of randomized controlled trials should use the integrated checklist presented in this paper.

A retrospective analysis of conditional power assumptions in clinical trials with continuous or binary endpoints

BACKGROUND

Adaptive clinical trials may use conditional power (CP) to make decisions at interim analyses, requiring assumptions about the treatment effect for remaining patients. It is critical that these assumptions are understood by those using CP in decision-making, as well as timings of these decisions.

METHODS

Data for 21 outcomes from 14 published clinical trials were made available for re-analysis. CP curves for accruing outcome information were calculated using and compared with a pre-specified objective criteria for original and transformed versions of the trial data using four future treatment effect assumptions: (i) observed current trend, (ii) hypothesised effect, (iii) 80% optimistic confidence limit, (iv) 90% optimistic confidence limit.

RESULTS

The hypothesised effect assumption met objective criteria when the true effect was close to that planned, but not when smaller than planned. The opposite was seen using the current trend assumption. Optimistic confidence limit assumptions appeared to offer a compromise between the two, performing well against objective criteria when the end observed effect was as planned or smaller.

CONCLUSION

The current trend assumption could be the preferable assumption when there is a wish to stop early for futility. Interim analyses could be undertaken as early as 30% of patients have data available. Optimistic confidence limit assumptions should be considered when using CP to make trial decisions, although later interim timings should be considered where logistically feasible.

Developing guidance for a risk-proportionate approach to blinding statisticians within clinical trials: a mixed methods study

BACKGROUND

Existing guidelines recommend statisticians remain blinded to treatment allocation prior to the final analysis and that any interim analyses should be conducted by a separate team from the one undertaking the final analysis. However, there remains substantial variation in practice between UK Clinical Trials Units (CTUs) when it comes to blinding statisticians. Therefore, the aim of this study was to develop guidance to advise CTUs on a risk-proportionate approach to blinding statisticians within clinical trials.

METHODS

This study employed a mixed methods approach involving three stages: (I) a quantitative study using a cohort of 200 studies (from a major UK funder published between 2016 and 2020) to assess the impact of blinding statisticians on the proportion of trials reporting a statistically significant finding for the primary outcome(s); (II) a qualitative study using focus groups to determine the perspectives of key stakeholders on the practice of blinding trial statisticians; and (III) combining the results of stages I and II, along with a stakeholder meeting, to develop guidance for UK CTUs.

RESULTS

After screening abstracts, 179 trials were included for review. The results of the primary analysis showed no evidence that involvement of an unblinded trial statistician was associated with the likelihood of statistically significant findings being reported, odds ratio (OR) 1.02 (95% confidence interval (CI) 0.49 to 2.13). Six focus groups were conducted, with 37 participants. The triangulation between stages I and II resulted in developing 40 provisional statements. These were rated independently by the stakeholder group prior to the meeting. Ten statements reached agreement with no agreement on 30 statements. At the meeting, various factors were identified that could influence the decision of blinding the statistician, including timing, study design, types of intervention and practicalities. Guidance including 21 recommendations/considerations was developed alongside a Risk Assessment Tool to provide CTUs with a framework for assessing the risks associated with blinding/not blinding statisticians and for identifying appropriate mitigation strategies.

CONCLUSIONS

This is the first study to develop a guidance document to enhance the understanding of blinding statisticians and to provide a framework for the decision-making process. The key finding was that the decision to blind statisticians should be based on the benefits and risks associated with a particular trial.

TRial to Assess Implementation of New research in a primary care Setting (TRAINS): study protocol for a pragmatic cluster randomised controlled trial of an educational intervention to promote asthma prescription uptake in general practitioner practices

Background

There is a marked increase in unscheduled care visits in school-aged children with asthma after returning to school in September. This is potentially associated with children not taking their asthma preventer medication during the school summer holidays. A cluster randomised controlled trial (PLEASANT) was undertaken with 1279 school-age children in 141 general practices (71 on intervention and 70 on control) in England and Wales. It found that a simple letter sent from the family doctor during the school holidays to a parent with a child with asthma, informing them of the importance of taking asthma preventer medication during the summer relatively increased prescriptions by 30% in August and reduced medical contacts in the period September to December. Also, it is estimated there was a cost-saving of £36.07 per patient over the year. We aim to conduct a randomised trial to assess if informing GP practices of an evidence-based intervention improves the implementation of that intervention.

Methods/design

The TRAINS study—TRial to Assess Implementation of New research in a primary care Setting—is a pragmatic cluster randomised implementation trial using routine data. A total of 1389 general practitioner (GP) practices in England will be included into the trial; 694 GP practices will be randomised to the intervention group and 695 control group of usual care. The Clinical Practice Research Datalink (CPRD) will send the intervention and obtain all data for the study, including prescription and primary care contacts data. The intervention will be sent in June 2021 by postal and email to the asthma lead and/or practice manager. The intervention is a letter to GPs informing them of the PLEASANT study findings with recommendations. It will come with an information leaflet about PLEASANT and a suggested reminder letter and SMS text template.

Discussion

The trial will assess if informing GP practices of the PLEASANT trial results will increase prescription uptake before the start of the school year. The hope is that the intervention will increase the implementation of PLEASANT work and then increase prescription uptake during the summer holiday prior to the start of school.

Trial registration

ClinicalTrials.gov ID: NCT05226091

Supplementary Information

The online version contains supplementary material available at 10.1186/s13063-022-06864-y.

Optimal pharmacotherapy pathway in adults with diabetic peripheral neuropathic pain: the OPTION-DM RCT

BACKGROUND

The mainstay of treatment for diabetic peripheral neuropathic pain is pharmacotherapy, but the current National Institute for Health and Care Excellence guideline is not based on robust evidence, as the treatments and their combinations have not been directly compared.

OBJECTIVES

To determine the most clinically beneficial, cost-effective and tolerated treatment pathway for diabetic peripheral neuropathic pain.

DESIGN

A randomised crossover trial with health economic analysis.

SETTING

Twenty-one secondary care centres in the UK.

PARTICIPANTS

Adults with diabetic peripheral neuropathic pain with a 7-day average self-rated pain score of ≥ 4 points (Numeric Rating Scale 0-10).

INTERVENTIONS

Participants were randomised to three commonly used treatment pathways: (1) amitriptyline supplemented with pregabalin, (2) duloxetine supplemented with pregabalin and (3) pregabalin supplemented with amitriptyline. Participants and research teams were blinded to treatment allocation, using over-encapsulated capsules and matching placebos. Site pharmacists were unblinded.

OUTCOMES

The primary outcome was the difference in 7-day average 24-hour Numeric Rating Scale score between pathways, measured during the final week of each pathway. Secondary end points included 7-day average daily Numeric Rating Scale pain score at week 6 between monotherapies, quality of life (Short Form questionnaire-36 items), Hospital Anxiety and Depression Scale score, the proportion of patients achieving 30% and 50% pain reduction, Brief Pain Inventory - Modified Short Form items scores, Insomnia Severity Index score, Neuropathic Pain Symptom Inventory score, tolerability (scale 0-10), Patient Global Impression of Change score at week 16 and patients' preferred treatment pathway at week 50. Adverse events and serious adverse events were recorded. A within-trial cost-utility analysis was carried out to compare treatment pathways using incremental costs per quality-adjusted life-years from an NHS and social care perspective.

RESULTS

A total of 140 participants were randomised from 13 UK centres, 130 of whom were included in the analyses. Pain score at week 16 was similar between the arms, with a mean difference of -0.1 points (98.3% confidence interval -0.5 to 0.3 points) for duloxetine supplemented with pregabalin compared with amitriptyline supplemented with pregabalin, a mean difference of -0.1 points (98.3% confidence interval -0.5 to 0.3 points) for pregabalin supplemented with amitriptyline compared with amitriptyline supplemented with pregabalin and a mean difference of 0.0 points (98.3% confidence interval -0.4 to 0.4 points) for pregabalin supplemented with amitriptyline compared with duloxetine supplemented with pregabalin. Results for tolerability, discontinuation and quality of life were similar. The adverse events were predictable for each drug. Combination therapy (weeks 6-16) was associated with a further reduction in Numeric Rating Scale pain score (mean 1.0 points, 98.3% confidence interval 0.6 to 1.3 points) compared with those who remained on monotherapy (mean 0.2 points, 98.3% confidence interval -0.1 to 0.5 points). The pregabalin supplemented with amitriptyline pathway had the fewest monotherapy discontinuations due to treatment-emergent adverse events and was most commonly preferred (most commonly preferred by participants: amitriptyline supplemented with pregabalin, 24%; duloxetine supplemented with pregabalin, 33%; pregabalin supplemented with amitriptyline, 43%; p = 0.26). No single pathway was superior in cost-effectiveness. The incremental gains in quality-adjusted life-years were small for each pathway comparison [amitriptyline supplemented with pregabalin compared with duloxetine supplemented with pregabalin -0.002 (95% confidence interval -0.011 to 0.007) quality-adjusted life-years, amitriptyline supplemented with pregabalin compared with pregabalin supplemented with amitriptyline -0.006 (95% confidence interval -0.002 to 0.014) quality-adjusted life-years and duloxetine supplemented with pregabalin compared with pregabalin supplemented with amitriptyline 0.007 (95% confidence interval 0.0002 to 0.015) quality-adjusted life-years] and incremental costs over 16 weeks were similar [amitriptyline supplemented with pregabalin compared with duloxetine supplemented with pregabalin -£113 (95% confidence interval -£381 to £90), amitriptyline supplemented with pregabalin compared with pregabalin supplemented with amitriptyline £155 (95% confidence interval -£37 to £625) and duloxetine supplemented with pregabalin compared with pregabalin supplemented with amitriptyline £141 (95% confidence interval -£13 to £398)].

LIMITATIONS

Although there was no placebo arm, there is strong evidence for the use of each study medication from randomised placebo-controlled trials. The addition of a placebo arm would have increased the duration of this already long and demanding trial and it was not felt to be ethically justifiable.

FUTURE WORK

Future research should explore (1) variations in diabetic peripheral neuropathic pain management at the practice level, (2) how OPTION-DM (Optimal Pathway for TreatIng neurOpathic paiN in Diabetes Mellitus) trial findings can be best implemented, (3) why some patients respond to a particular drug and others do not and (4) what options there are for further treatments for those patients on combination treatment with inadequate pain relief.

CONCLUSIONS

The three treatment pathways appear to give comparable patient outcomes at similar costs, suggesting that the optimal treatment may depend on patients' preference in terms of side effects.

TRIAL REGISTRATION

The trial is registered as ISRCTN17545443 and EudraCT 2016-003146-89.

FUNDING

This project was funded by the National Institute for Health and Care Research (NIHR) Health Technology Assessment programme, and will be published in full in Health Technology Assessment; Vol. 26, No. 39. See the NIHR Journals Library website for further project information.

Practical guide to sample size calculations: Installation of the app SampSize

In 2016 we published three articles in Pharmaceutical Statistics that gave a practical guide to sample size calculations. In each of the articles there were instructions on how to obtain the App SampSize. This short communication updates these instructions and highlights the updates and added functionality to the App.

Adjusting for bias in the mean for primary and secondary outcomes when trials are in sequence

When designing a clinical trial, one key aspect of the design is the sample size calculation. The sample size calculation tends to rely on a target or expected difference. The expected difference can be based on the observed data from previous studies, which results in bias. It has been reported that large treatment effects observed in trials are often not replicated in subsequent trials. If these values are used to design subsequent studies, the sample sizes may be biased which results in an unethical study. Regression to the mean (RTM) is one explanation for this. If only health technologies which meet a particular continuation criterion (such as p < 0.05 in the first study) are progressed to a second confirmatory trial, it is highly likely that the observed effect in the second trial will be lower than that observed in the first trial. It will be shown how when moving from one trial to the next, a truncated normal distribution is inherently imposed on the first study. This results in a lower observed effect size in the second trial. A simple adjustment method is proposed based on the mathematical properties of the truncated normal distribution. This adjustment method was confirmed using simulations in R and compared with other previous adjustments. The method can be applied to the observed effect in a trial, which is being used in the design of a second confirmatory trial, resulting in a more stable estimate for the 'true' treatment effect. The adjustment accounts for the bias in the primary and secondary endpoints in the first trial with the bias being affected by the power of that study. Tables of results have been provided to aid implementation, along with a worked example. In summary, there is a bias introduced when the point estimate from one trial is used to assist the design of a second trial. It is recommended that any observed point estimates be used with caution and the adjustment method developed in this article be implemented to significantly reduce this bias.

Sample sizes for cluster-randomised trials with continuous outcomes: Accounting for uncertainty in a single intra-cluster correlation estimate

Sample size calculations for cluster-randomised trials require inclusion of an inflation factor taking into account the intra-cluster correlation coefficient. Often, estimates of the intra-cluster correlation coefficient are taken from pilot trials, which are known to have uncertainty about their estimation. Given that the value of the intra-cluster correlation coefficient has a considerable influence on the calculated sample size for a main trial, the uncertainty in the estimate can have a large impact on the ultimate sample size and consequently, the power of a main trial. As such, it is important to account for the uncertainty in the estimate of the intra-cluster correlation coefficient. While a commonly adopted approach is to utilise the upper confidence limit in the sample size calculation, this is a largely inefficient method which can result in overpowered main trials. In this paper, we present a method of estimating the sample size for a main cluster-randomised trial with a continuous outcome, using numerical methods to account for the uncertainty in the intra-cluster correlation coefficient estimate. Despite limitations with this initial study, the findings and recommendations in this paper can help to improve sample size estimations for cluster randomised controlled trials by accounting for uncertainty in the estimate of the intra-cluster correlation coefficient. We recommend this approach be applied to all trials where there is uncertainty in the intra-cluster correlation coefficient estimate, in conjunction with additional sources of information to guide the estimation of the intra-cluster correlation coefficient.

A systematic review of the "promising zone" design

Introduction

Sample size calculations require assumptions regarding treatment response and variability. Incorrect assumptions can result in under- or overpowered trials, posing ethical concerns. Sample size re-estimation (SSR) methods investigate the validity of these assumptions and increase the sample size if necessary. The “promising zone” (Mehta and Pocock, Stat Med 30:3267–3284, 2011) concept is appealing to researchers for its design simplicity. However, it is still relatively new in the application and has been a source of controversy.

Objectives

This research aims to synthesise current approaches and practical implementation of the promising zone design.

Methods

This systematic review comprehensively identifies the reporting of methodological research and of clinical trials using promising zone. Databases were searched according to a pre-specified search strategy, and pearl growing techniques implemented.

Results

The combined search methods resulted in 270 unique records identified; 171 were included in the review, of which 30 were trials. The median time to the interim analysis was 60% of the original target sample size (IQR 41–73%). Of the 15 completed trials, 7 increased their sample size. Only 21 studies reported the maximum sample size that would be considered, for which the median increase was 50% (IQR 35–100%).

Conclusions

Promising zone is being implemented in a range of trials worldwide, albeit in low numbers. Identifying trials using promising zone was difficult due to the lack of reporting of SSR methodology. Even when SSR methodology was reported, some had key interim analysis details missing, and only eight papers provided promising zone ranges.

The adaptive designs CONSORT extension (ACE) statement: a checklist with explanation and elaboration guideline for reporting randomised trials that use an adaptive design

Adaptive designs (ADs) allow pre-planned changes to an ongoing trial without compromising the validity of conclusions and it is essential to distinguish pre-planned from unplanned changes that may also occur. The reporting of ADs in randomised trials is inconsistent and needs improving. Incompletely reported AD randomised trials are difficult to reproduce and are hard to interpret and synthesise. This consequently hampers their ability to inform practice as well as future research and contributes to research waste. Better transparency and adequate reporting will enable the potential benefits of ADs to be realised.This extension to the Consolidated Standards Of Reporting Trials (CONSORT) 2010 statement was developed to enhance the reporting of randomised AD clinical trials. We developed an Adaptive designs CONSORT Extension (ACE) guideline through a two-stage Delphi process with input from multidisciplinary key stakeholders in clinical trials research in the public and private sectors from 21 countries, followed by a consensus meeting. Members of the CONSORT Group were involved during the development process.The paper presents the ACE checklists for AD randomised trial reports and abstracts, as well as an explanation with examples to aid the application of the guideline. The ACE checklist comprises seven new items, nine modified items, six unchanged items for which additional explanatory text clarifies further considerations for ADs, and 20 unchanged items not requiring further explanatory text. The ACE abstract checklist has one new item, one modified item, one unchanged item with additional explanatory text for ADs, and 15 unchanged items not requiring further explanatory text.The intention is to enhance transparency and improve reporting of AD randomised trials to improve the interpretability of their results and reproducibility of their methods, results and inference. We also hope indirectly to facilitate the much-needed knowledge transfer of innovative trial designs to maximise their potential benefits. In order to encourage its wide dissemination this article is freely accessible on the BMJ and Trials journal websites."To maximise the benefit to society, you need to not just do research but do it well" Douglas G Altman.

The Adaptive designs CONSORT Extension (ACE) statement: a checklist with explanation and elaboration guideline for reporting randomised trials that use an adaptive design

Adaptive designs (ADs) allow pre-planned changes to an ongoing trial without compromising the validity of conclusions and it is essential to distinguish pre-planned from unplanned changes that may also occur. The reporting of ADs in randomised trials is inconsistent and needs improving. Incompletely reported AD randomised trials are difficult to reproduce and are hard to interpret and synthesise. This consequently hampers their ability to inform practice as well as future research and contributes to research waste. Better transparency and adequate reporting will enable the potential benefits of ADs to be realised.This extension to the Consolidated Standards Of Reporting Trials (CONSORT) 2010 statement was developed to enhance the reporting of randomised AD clinical trials. We developed an Adaptive designs CONSORT Extension (ACE) guideline through a two-stage Delphi process with input from multidisciplinary key stakeholders in clinical trials research in the public and private sectors from 21 countries, followed by a consensus meeting. Members of the CONSORT Group were involved during the development process.The paper presents the ACE checklists for AD randomised trial reports and abstracts, as well as an explanation with examples to aid the application of the guideline. The ACE checklist comprises seven new items, nine modified items, six unchanged items for which additional explanatory text clarifies further considerations for ADs, and 20 unchanged items not requiring further explanatory text. The ACE abstract checklist has one new item, one modified item, one unchanged item with additional explanatory text for ADs, and 15 unchanged items not requiring further explanatory text.The intention is to enhance transparency and improve reporting of AD randomised trials to improve the interpretability of their results and reproducibility of their methods, results and inference. We also hope indirectly to facilitate the much-needed knowledge transfer of innovative trial designs to maximise their potential benefits.

Practical help for specifying the target difference in sample size calculations for RCTs: the DELTA2 five-stage study, including a workshop

BACKGROUND

The randomised controlled trial is widely considered to be the gold standard study for comparing the effectiveness of health interventions. Central to its design is a calculation of the number of participants needed (the sample size) for the trial. The sample size is typically calculated by specifying the magnitude of the difference in the primary outcome between the intervention effects for the population of interest. This difference is called the 'target difference' and should be appropriate for the principal estimand of interest and determined by the primary aim of the study. The target difference between treatments should be considered realistic and/or important by one or more key stakeholder groups.

OBJECTIVE

The objective of the report is to provide practical help on the choice of target difference used in the sample size calculation for a randomised controlled trial for researchers and funder representatives.

METHODS

The Difference ELicitation in TriAls2 (DELTA2) recommendations and advice were developed through a five-stage process, which included two literature reviews of existing funder guidance and recent methodological literature; a Delphi process to engage with a wider group of stakeholders; a 2-day workshop; and finalising the core document.

RESULTS

Advice is provided for definitive trials (Phase III/IV studies). Methods for choosing the target difference are reviewed. To aid those new to the topic, and to encourage better practice, 10 recommendations are made regarding choosing the target difference and undertaking a sample size calculation. Recommended reporting items for trial proposal, protocols and results papers under the conventional approach are also provided. Case studies reflecting different trial designs and covering different conditions are provided. Alternative trial designs and methods for choosing the sample size are also briefly considered.

CONCLUSIONS

Choosing an appropriate sample size is crucial if a study is to inform clinical practice. The number of patients recruited into the trial needs to be sufficient to answer the objectives; however, the number should not be higher than necessary to avoid unnecessary burden on patients and wasting precious resources. The choice of the target difference is a key part of this process under the conventional approach to sample size calculations. This document provides advice and recommendations to improve practice and reporting regarding this aspect of trial design. Future work could extend the work to address other less common approaches to the sample size calculations, particularly in terms of appropriate reporting items.

FUNDING

Funded by the Medical Research Council (MRC) UK and the National Institute for Health Research as part of the MRC-National Institute for Health Research Methodology Research programme.

Progression criteria in trials with an internal pilot: an audit of publicly funded randomised controlled trials

Background

With millions of pounds spent annually on medical research in the UK, it is important that studies are spending funds wisely. Internal pilots offer the chance to stop a trial early if it becomes apparent that the study will not be able to recruit enough patients to show whether an intervention is clinically effective. This study aims to assess the use of internal pilots in individually randomised controlled trials funded by the Health Technology Assessment (HTA) programme and to summarise the progression criteria chosen in these trials.

Methods

Studies were identified from reports of the HTA committees’ funding decisions from 2012 to 2016. In total, 242 trials were identified of which 134 were eligible to be included in the audit. Protocols for the eligible studies were located on the NIHR Journals website, and if protocols were not available online then study managers were contacted to provide information.

Results

Over two-thirds (72.4%) of studies said in their protocol that they would include an internal pilot phase for their study and 37.8% of studies without an internal pilot had done an external pilot study to assess the feasibility of the full study. A typical study with an internal pilot has a target sample size of 510 over 24 months and aims to recruit one-fifth of their total target sample size within the first one-third of their recruitment time.

There has been an increase in studies adopting a three-tiered structure for their progression rules in recent years, with 61.5% (16/26) of studies using the system in 2016 compared to just 11.8% (2/17) in 2015. There was also a rise in the number of studies giving a target recruitment rate in their progression criteria: 42.3% (11/26) in 2016 compared to 35.3% (6/17) in 2015.

Conclusions

Progression criteria for an internal pilot are usually well specified but targets vary widely. For the actual criteria, red/amber/green systems have increased in popularity in recent years. Trials should justify the targets they have set, especially where targets are low.

Development process of a consensus-driven CONSORT extension for randomised trials using an adaptive design

BACKGROUND

Adequate reporting of adaptive designs (ADs) maximises their potential benefits in the conduct of clinical trials. Transparent reporting can help address some obstacles and concerns relating to the use of ADs. Currently, there are deficiencies in the reporting of AD trials. To overcome this, we have developed a consensus-driven extension to the CONSORT statement for randomised trials using an AD. This paper describes the processes and methods used to develop this extension rather than detailed explanation of the guideline.

METHODS

We developed the guideline in seven overlapping stages: 1) Building on prior research to inform the need for a guideline; 2) A scoping literature review to inform future stages; 3) Drafting the first checklist version involving an External Expert Panel; 4) A two-round Delphi process involving international, multidisciplinary, and cross-sector key stakeholders; 5) A consensus meeting to advise which reporting items to retain through voting, and to discuss the structure of what to include in the supporting explanation and elaboration (E&E) document; 6) Refining and finalising the checklist; and 7) Writing-up and dissemination of the E&E document. The CONSORT Executive Group oversaw the entire development process.

RESULTS

Delphi survey response rates were 94/143 (66%), 114/156 (73%), and 79/143 (55%) in rounds 1, 2, and across both rounds, respectively. Twenty-seven delegates from Europe, the USA, and Asia attended the consensus meeting. The main checklist has seven new and nine modified items and six unchanged items with expanded E&E text to clarify further considerations for ADs. The abstract checklist has one new and one modified item together with an unchanged item with expanded E&E text. The E&E document will describe the scope of the guideline, the definition of an AD, and some types of ADs and trial adaptations and explain each reporting item in detail including case studies.

CONCLUSIONS

We hope that making the development processes, methods, and all supporting information that aided decision-making transparent will enhance the acceptability and quick uptake of the guideline. This will also help other groups when developing similar CONSORT extensions. The guideline is applicable to all randomised trials with an AD and contains minimum reporting requirements.

Calculation of confidence intervals for a finite population size

For any estimate of response, confidence intervals are important as they help quantify a plausible range of values for the population response. However, there may be instances in clinical research when the population size is finite, but we wish to take a sample from the population and make inference from this sample. Instances where you can have a fixed population size include when undertaking a clinical audit of patient records or in a clinical trial a researcher could be checking for transcription errors against patient notes. In this paper, we describe how confidence interval calculations can be calculated for a finite population. These confidence intervals are narrower than confidence intervals from population samples. For the extreme case of when a 100% sample from the population is taken, there is no error and the calculation is the population response. The methods in the paper are described using a case study from clinical data management.

DELTA2 guidance on choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial

Background

A key step in the design of a RCT is the estimation of the number of participants needed in the study. The most common approach is to specify a target difference between the treatments for the primary outcome and then calculate the required sample size. The sample size is chosen to ensure that the trial will have a high probability (adequate statistical power) of detecting a target difference between the treatments should one exist.

The sample size has many implications for the conduct and interpretation of the study. Despite the critical role that the target difference has in the design of a RCT, the way in which it is determined has received little attention. In this article, we summarise the key considerations and messages from new guidance for researchers and funders on specifying the target difference, and undertaking and reporting a RCT sample size calculation. This article on choosing the target difference for a randomised controlled trial (RCT) and undertaking and reporting the sample size calculation has been dual published in the BMJ and BMC Trials journals

Methods

The DELTA² (Difference ELicitation in TriAls) project comprised five major components: systematic literature reviews of recent methodological developments (stage 1) and existing funder guidance (stage 2); a Delphi study (stage 3); a two-day consensus meeting bringing together researchers, funders and patient representatives (stage 4); and the preparation and dissemination of a guidance document (stage 5).

Results and Discussion

The key messages from the DELTA² guidance on determining the target difference and sample size calculation for a randomised caontrolled trial are presented. Recommendations for the subsequent reporting of the sample size calculation are also provided.

DELTA2 guidance on choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial

BACKGROUND

A key step in the design of a RCT is the estimation of the number of participants needed in the study. The most common approach is to specify a target difference between the treatments for the primary outcome and then calculate the required sample size. The sample size is chosen to ensure that the trial will have a high probability (adequate statistical power) of detecting a target difference between the treatments should one exist. The sample size has many implications for the conduct and interpretation of the study. Despite the critical role that the target difference has in the design of a RCT, the way in which it is determined has received little attention. In this article, we summarise the key considerations and messages from new guidance for researchers and funders on specifying the target difference, and undertaking and reporting a RCT sample size calculation. This article on choosing the target difference for a randomised controlled trial (RCT) and undertaking and reporting the sample size calculation has been dual published in the BMJ and BMC Trials journals METHODS: The DELTA² (Difference ELicitation in TriAls) project comprised five major components: systematic literature reviews of recent methodological developments (stage 1) and existing funder guidance (stage 2); a Delphi study (stage 3); a two-day consensus meeting bringing together researchers, funders and patient representatives (stage 4); and the preparation and dissemination of a guidance document (stage 5).

RESULTS AND DISCUSSION

The key messages from the DELTA² guidance on determining the target difference and sample size calculation for a randomised caontrolled trial are presented. Recommendations for the subsequent reporting of the sample size calculation are also provided.

Choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial - the development of the DELTA2 guidance

BACKGROUND

A key step in the design of a randomised controlled trial is the estimation of the number of participants needed. The most common approach is to specify a target difference in the primary outcome between the randomised groups and then estimate the corresponding sample size. The sample size is chosen to provide reassurance that the trial will have high statistical power to detect the target difference at the planned statistical significance level. Alternative approaches are also available, though most still require specification of a target difference. The sample size has many implications for the conduct of the study, as well as incurring scientific and ethical aspects. Despite the critical role of the target difference for the primary outcome in the design of a randomised controlled trial (RCT), the manner in which it is determined has received little attention. This article reports the development of the DELTA² guidance on the specification and reporting of the target difference for the primary outcome in a sample size calculation for a RCT.

METHODS

The DELTA² (Difference ELicitation in TriAls) project has five components comprising systematic literature reviews of recent methodological developments (stage 1) and existing funder guidance (stage 2), a Delphi study (stage 3), a 2-day consensus meeting bringing together researchers, funders and patient representatives (stage 4), and the preparation and dissemination of a guidance document (stage 5).

RESULTS

The project started in April 2016. The literature search identified 28 articles of methodological developments relevant to a method for specifying a target difference. A Delphi study involving 69 participants, along with a 2-day consensus meeting were conducted. In addition, further engagement sessions were held at two international conferences. The main guidance text was finalised on April 18, 2018, after revision informed by feedback gathered from stages 2 and 3 and from funder representatives.

DISCUSSION

The DELTA² Delphi study identified a number of areas (such as practical recommendations and examples, greater coverage of different trial designs and statistical approaches) of particular interest amongst stakeholders which new guidance was desired to meet. New relevant references were identified by the review. Such findings influenced the scope, drafting and revision of the guidance. While not all suggestions could be accommodated, it is hoped that the process has led to a more useful and practical document.

A study of target effect sizes in randomised controlled trials published in the Health Technology Assessment journal

BACKGROUND

When designing a randomised controlled trial (RCT), an important consideration is the sample size required. This is calculated from several components; one of which is the target difference. This study aims to review the currently reported methods of elicitation of the target difference as well as to quantify the target differences used in Health Technology Assessment (HTA)-funded trials.

METHODS

Trials were identified from the National Institute of Health Research Health Technology Assessment journal. A total of 177 RCTs published between 2006 and 2016 were assessed for eligibility. Eligibility was established by the design of the trial and the quality of data available. The trial designs were parallel-group, superiority RCTs with a continuous primary endpoint. Data were extracted and the standardised anticipated and observed effect size estimates were calculated. Exclusion criteria was based on trials not providing enough detail in the sample size calculation and results, and trials not being of parallel-group, superiority design.

RESULTS

A total of 107 RCTs were included in the study from 102 reports. The most commonly reported method for effect size derivation was a review of evidence and use of previous research (52.3%). This was common across all clinical areas. The median standardised target effect size was 0.30 (interquartile range: 0.20-0.38), with the median standardised observed effect size 0.11 (IQR 0.05-0.29). The maximum anticipated and observed effect sizes were 0.76 and 1.18, respectively. Only two trials had anticipated target values above 0.60.

CONCLUSION

The most commonly reported method of elicitation of the target effect size is previous published research. The average target effect size was 0.3. A clear distinction between the target difference and the minimum clinically important difference is recommended when designing a trial. Transparent explanation of target difference elicitation is advised, with multiple methods including a review of evidence and opinion-seeking advised as the more optimal methods for effect size quantification.

Open-label, cluster randomised controlled trial and economic evaluation of a brief letter from a GP on unscheduled medical contacts associated with the start of the school year: the PLEASANT trial

BACKGROUND

Asthma is seasonal with peaks in exacerbation rates in school-age children associated with the return to school following the summer vacation. A drop in prescription collection in August is associated with an increase in the number of unscheduled contacts after the school return.

OBJECTIVE

To assess whether a public health intervention delivered in general practice reduced unscheduled medical contacts in children with asthma.

DESIGN

Cluster randomised trial with trial-based economic evaluation. Randomisation was at general practice level, stratified by size of practice. The intervention group received a letter from their general practitioner (GP) in late July outlining the importance of (re)taking asthma medication before the return to school. The control group was usual care.

SETTING

General practices in England and Wales.

PARTICIPANTS

12 179 school-age children in 142 general practices (70 randomised to intervention).

MAIN OUTCOME

Proportion of children aged 5-16 years who had an unscheduled contact in September. Secondary endpoints included collection of prescriptions in August and medical contacts over 12 months (September-August). Economic endpoints were quality-adjusted life-years gained and health service costs.

RESULTS

There was no evidence of effect (OR 1.09; 95% CI 0.96 to 1.25 against treatment) on unscheduled contacts in September. The intervention increased the proportion of children collecting a prescription in August by 4% (OR 1.43; 95% CI 1.24 to 1.64). The intervention also reduced the total number of medical contacts between September-August by 5% (incidence ratio 0.95; 95% CI 0.91 to 0.99).The mean reduction in medical contacts informed the health economics analyses. The intervention was estimated to save £36.07 per patient, with a high probability (96.3%) of being cost-saving.

CONCLUSIONS

The intervention succeeded in increasing children collecting prescriptions. It did not reduce unscheduled care in September (the primary outcome), but in the year following the intervention, it reduced the total number of medical contacts.

TRIAL REGISTRATION NUMBER

ISRCTN03000938; Results.

Are pilot trials useful for predicting randomisation and attrition rates in definitive studies: A review of publicly funded trials

BACKGROUND/AIMS

External pilot trials are recommended for testing the feasibility of main or confirmatory trials. However, there is little evidence that progress in external pilot trials actually predicts randomisation and attrition rates in the main trial. To assess the use of external pilot trials in trial design, we compared randomisation and attrition rates in publicly funded randomised controlled trials with rates in their pilots.

METHODS

Randomised controlled trials for which there was an external pilot trial were identified from reports published between 2004 and 2013 in the Health Technology Assessment Journal. Data were extracted from published papers, protocols and reports. Bland-Altman plots and descriptive statistics were used to investigate the agreement of randomisation and attrition rates between the full and external pilot trials.

RESULTS

Of 561 reports, 41 were randomised controlled trials with pilot trials and 16 met criteria for a pilot trial with sufficient data. Mean attrition and randomisation rates were 21.1% and 50.4%, respectively, in the pilot trials and 16.8% and 65.2% in the main. There was minimal bias in the pilot trial when predicting the main trial attrition and randomisation rate. However, the variation was large: the mean difference in the attrition rate between the pilot and main trial was -4.4% with limits of agreement of -37.1% to 28.2%. Limits of agreement for randomisation rates were -47.8% to 77.5%.

CONCLUSION

Results from external pilot trials to estimate randomisation and attrition rates should be used with caution as comparison of the difference in the rates between pilots and their associated full trial demonstrates high variability. We suggest using internal pilot trials wherever appropriate.

PLEASANT: Preventing and Lessening Exacerbations of Asthma in School-age children Associated with a New Term - a cluster randomised controlled trial and economic evaluation

BACKGROUND

Asthma episodes and deaths are known to be seasonal. A number of reports have shown peaks in asthma episodes in school-aged children associated with the return to school following the summer vacation. A fall in prescription collection in the month of August has been observed, and was associated with an increase in the number of unscheduled contacts after the return to school in September.

OBJECTIVE

The primary objective of the study was to assess whether or not a NHS-delivered public health intervention reduces the September peak in unscheduled medical contacts.

DESIGN

Cluster randomised trial, with the unit of randomisation being 142 NHS general practices, and trial-based economic evaluation.

SETTING

Primary care.

INTERVENTION

A letter sent (n = 70 practices) in July from their general practitioner (GP) to parents/carers of school-aged children with asthma to remind them of the importance of taking their medication, and to ensure that they have sufficient medication prior to the start of the new school year in September. The control group received usual care.

MAIN OUTCOME MEASURES

The primary outcome measure was the proportion of children aged 5-16 years who had an unscheduled medical contact in September 2013. Supporting end points included the proportion of children who collected prescriptions in August 2013 and unscheduled contacts through the following 12 months. Economic end points were quality-adjusted life-years (QALYs) gained and costs from an NHS and Personal Social Services perspective.

RESULTS

There is no evidence of effect in terms of unscheduled contacts in September. Among children aged 5-16 years, the odds ratio (OR) was 1.09 [95% confidence interval (CI) 0.96 to 1.25] against the intervention. The intervention did increase the proportion of children collecting a prescription in August (OR 1.43, 95% CI 1.24 to 1.64) as well as scheduled contacts in the same month (OR 1.13, 95% CI 0.84 to 1.52). For the wider time intervals (September-December 2013 and September-August 2014), there is weak evidence of the intervention reducing unscheduled contacts. The intervention did not reduce unscheduled care in September, although it succeeded in increasing the proportion of children collecting prescriptions in August as well as having scheduled contacts in the same month. These unscheduled contacts in September could be a result of the intervention, as GPs may have wanted to see patients before issuing a prescription. The economic analysis estimated a high probability that the intervention was cost-saving, for baseline-adjusted costs, across both base-case and sensitivity analyses. There was no increase in QALYs.

LIMITATION

The use of routine data led to uncertainty in the coding of medical contacts. The uncertainty was mitigated by advice from a GP adjudication panel.

CONCLUSIONS

The intervention did not reduce unscheduled care in September, although it succeeded in increasing the proportion of children both collecting prescriptions and having scheduled contacts in August. After September there is weak evidence in favour of the intervention. The intervention had a favourable impact on costs but did not demonstrate any impact on QALYs. The results of the trial indicate that further work is required on assessing and understanding adherence, both in terms of using routine data to make quantitative assessments, and through additional qualitative interviews with key stakeholders such as practice nurses, GPs and a wider group of children with asthma.

TRIAL REGISTRATION

Current Controlled Trials ISRCTN03000938.

FUNDING DETAILS

This project was funded by the NIHR Health Technology Assessment programme and will be published in full in Health Technology Assessment; Vol. 20, No. 93. See the HTA programme website for further project information.

Guidance for using pilot studies to inform the design of intervention trials with continuous outcomes

Background

A pilot study can be an important step in the assessment of an intervention by providing information to design the future definitive trial. Pilot studies can be used to estimate the recruitment and retention rates and population variance and to provide preliminary evidence of efficacy potential. However, estimation is poor because pilot studies are small, so sensitivity analyses for the main trial’s sample size calculations should be undertaken.

Methods

We demonstrate how to carry out easy-to-perform sensitivity analysis for designing trials based on pilot data using an example. Furthermore, we introduce rules of thumb for the size of the pilot study so that the overall sample size, for both pilot and main trials, is minimized.

Results

The example illustrates how sample size estimates for the main trial can alter dramatically by plausibly varying assumptions. Required sample size for 90% power varied from 392 to 692 depending on assumptions. Some scenarios were not feasible based on the pilot study recruitment and retention rates.

Conclusion

Pilot studies can be used to help design the main trial, but caution should be exercised. We recommend the use of sensitivity analyses to assess the robustness of the design assumptions for a main trial.

Design considerations and analysis planning of a phase 2a proof of concept study in rheumatoid arthritis in the presence of possible non-monotonicity

Background

It is important to quantify the dose response for a drug in phase 2a clinical trials so the optimal doses can then be selected for subsequent late phase trials. In a phase 2a clinical trial of new lead drug being developed for the treatment of rheumatoid arthritis (RA), a U-shaped dose response curve was observed. In the light of this result further research was undertaken to design an efficient phase 2a proof of concept (PoC) trial for a follow-on compound using the lessons learnt from the lead compound.

Methods

The planned analysis for the Phase 2a trial for GSK123456 was a Bayesian Emax model which assumes the dose-response relationship follows a monotonic sigmoid “S” shaped curve. This model was found to be suboptimal to model the U-shaped dose response observed in the data from this trial and alternatives approaches were needed to be considered for the next compound for which a Normal dynamic linear model (NDLM) is proposed. This paper compares the statistical properties of the Bayesian Emax model and NDLM model and both models are evaluated using simulation in the context of adaptive Phase 2a PoC design under a variety of assumed dose response curves: linear, Emax model, U-shaped model, and flat response.

Results

It is shown that the NDLM method is flexible and can handle a wide variety of dose-responses, including monotonic and non-monotonic relationships. In comparison to the NDLM model the Emax model excelled with higher probability of selecting ED90 and smaller average sample size, when the true dose response followed Emax like curve. In addition, the type I error, probability of incorrectly concluding a drug may work when it does not, is inflated with the Bayesian NDLM model in all scenarios which would represent a development risk to pharmaceutical company.

The bias, which is the difference between the estimated effect from the Emax and NDLM models and the simulated value, is comparable if the true dose response follows a placebo like curve, an Emax like curve, or log linear shape curve under fixed dose allocation, no adaptive allocation, half adaptive and adaptive scenarios. The bias though is significantly increased for the Emax model if the true dose response follows a U-shaped curve.

Conclusions

In most cases the Bayesian Emax model works effectively and efficiently, with low bias and good probability of success in case of monotonic dose response. However, if there is a belief that the dose response could be non-monotonic then the NDLM is the superior model to assess the dose response.

Electronic supplementary material

The online version of this article (10.1186/s12874-017-0416-3) contains supplementary material, which is available to authorized users.

Choosing the target difference ('effect size') for a randomised controlled trial - DELTA2 guidance protocol

BACKGROUND

A key step in the design of a randomised controlled trial (RCT) is the estimation of the number of participants needed. By far the most common approach is to specify a target difference and then estimate the corresponding sample size; this sample size is chosen to provide reassurance that the trial will have high statistical power to detect such a difference between the randomised groups (at the planned statistical significance level). The sample size has many implications for the conduct of the study, as well as carrying scientific and ethical aspects to its choice. Despite the critical role of the target difference for the primary outcome in the design of an RCT, the manner in which it is determined has received little attention. This article reports the protocol of the Difference ELicitation in TriAls (DELTA2) project, which will produce guidance on the specification and reporting of the target difference for the primary outcome in a sample size calculation for RCTs.

METHODS/DESIGN

The DELTA2 project has five components: systematic literature reviews of recent methodological developments (stage 1) and existing funder guidance (stage 2); a Delphi study (stage 3); a 2-day consensus meeting bringing together researchers, funders and patient representatives, as well as one-off engagement sessions at relevant stakeholder meetings (stage 4); and the preparation and dissemination of a guidance document (stage 5).

DISCUSSION

Specification of the target difference for the primary outcome is a key component of the design of an RCT. There is a need for better guidance for researchers and funders regarding specification and reporting of this aspect of trial design. The aim of this project is to produce consensus based guidance for researchers and funders.

Recruitment and retention of participants in randomised controlled trials: a review of trials funded and published by the United Kingdom Health Technology Assessment Programme

BACKGROUND

Substantial amounts of public funds are invested in health research worldwide. Publicly funded randomised controlled trials (RCTs) often recruit participants at a slower than anticipated rate. Many trials fail to reach their planned sample size within the envisaged trial timescale and trial funding envelope.

OBJECTIVES

To review the consent, recruitment and retention rates for single and multicentre randomised control trials funded and published by the UK's National Institute for Health Research (NIHR) Health Technology Assessment (HTA) Programme.

DATA SOURCES AND STUDY SELECTION

HTA reports of individually randomised single or multicentre RCTs published from the start of 2004 to the end of April 2016 were reviewed.

DATA EXTRACTION

Information was extracted, relating to the trial characteristics, sample size, recruitment and retention by two independent reviewers.

MAIN OUTCOME MEASURES

Target sample size and whether it was achieved; recruitment rates (number of participants recruited per centre per month) and retention rates (randomised participants retained and assessed with valid primary outcome data).

RESULTS

This review identified 151 individually RCTs from 787 NIHR HTA reports. The final recruitment target sample size was achieved in 56% (85/151) of the RCTs and more than 80% of the final target sample size was achieved for 79% of the RCTs (119/151). The median recruitment rate (participants per centre per month) was found to be 0.92 (IQR 0.43-2.79) and the median retention rate (proportion of participants with valid primary outcome data at follow-up) was estimated at 89% (IQR 79-97%).

CONCLUSIONS

There is considerable variation in the consent, recruitment and retention rates in publicly funded RCTs. Investigators should bear this in mind at the planning stage of their study and not be overly optimistic about their recruitment projections.

Automated telephone communication systems for preventive healthcare and management of long-term conditions

BACKGROUND

Automated telephone communication systems (ATCS) can deliver voice messages and collect health-related information from patients using either their telephone's touch-tone keypad or voice recognition software. ATCS can supplement or replace telephone contact between health professionals and patients. There are four different types of ATCS: unidirectional (one-way, non-interactive voice communication), interactive voice response (IVR) systems, ATCS with additional functions such as access to an expert to request advice (ATCS Plus) and multimodal ATCS, where the calls are delivered as part of a multicomponent intervention.

OBJECTIVES

To assess the effects of ATCS for preventing disease and managing long-term conditions on behavioural change, clinical, process, cognitive, patient-centred and adverse outcomes.

SEARCH METHODS

We searched 10 electronic databases (the Cochrane Central Register of Controlled Trials; MEDLINE; Embase; PsycINFO; CINAHL; Global Health; WHOLIS; LILACS; Web of Science; and ASSIA); three grey literature sources (Dissertation Abstracts, Index to Theses, Australasian Digital Theses); and two trial registries (www.controlled-trials.com; www.clinicaltrials.gov) for papers published between 1980 and June 2015.

SELECTION CRITERIA

Randomised, cluster- and quasi-randomised trials, interrupted time series and controlled before-and-after studies comparing ATCS interventions, with any control or another ATCS type were eligible for inclusion. Studies in all settings, for all consumers/carers, in any preventive healthcare or long term condition management role were eligible.

DATA COLLECTION AND ANALYSIS

We used standard Cochrane methods to select and extract data and to appraise eligible studies.

MAIN RESULTS

We included 132 trials (N = 4,669,689). Studies spanned across several clinical areas, assessing many comparisons based on evaluation of different ATCS types and variable comparison groups. Forty-one studies evaluated ATCS for delivering preventive healthcare, 84 for managing long-term conditions, and seven studies for appointment reminders. We downgraded our certainty in the evidence primarily because of the risk of bias for many outcomes. We judged the risk of bias arising from allocation processes to be low for just over half the studies and unclear for the remainder. We considered most studies to be at unclear risk of performance or detection bias due to blinding, while only 16% of studies were at low risk. We generally judged the risk of bias due to missing data and selective outcome reporting to be unclear.For preventive healthcare, ATCS (ATCS Plus, IVR, unidirectional) probably increase immunisation uptake in children (risk ratio (RR) 1.25, 95% confidence interval (CI) 1.18 to 1.32; 5 studies, N = 10,454; moderate certainty) and to a lesser extent in adolescents (RR 1.06, 95% CI 1.02 to 1.11; 2 studies, N = 5725; moderate certainty). The effects of ATCS in adults are unclear (RR 2.18, 95% CI 0.53 to 9.02; 2 studies, N = 1743; very low certainty).For screening, multimodal ATCS increase uptake of screening for breast cancer (RR 2.17, 95% CI 1.55 to 3.04; 2 studies, N = 462; high certainty) and colorectal cancer (CRC) (RR 2.19, 95% CI 1.88 to 2.55; 3 studies, N = 1013; high certainty) versus usual care. It may also increase osteoporosis screening. ATCS Plus interventions probably slightly increase cervical cancer screening (moderate certainty), but effects on osteoporosis screening are uncertain. IVR systems probably increase CRC screening at 6 months (RR 1.36, 95% CI 1.25 to 1.48; 2 studies, N = 16,915; moderate certainty) but not at 9 to 12 months, with probably little or no effect of IVR (RR 1.05, 95% CI 0.99, 1.11; 2 studies, 2599 participants; moderate certainty) or unidirectional ATCS on breast cancer screening.Appointment reminders delivered through IVR or unidirectional ATCS may improve attendance rates compared with no calls (low certainty). For long-term management, medication or laboratory test adherence provided the most general evidence across conditions (25 studies, data not combined). Multimodal ATCS versus usual care showed conflicting effects (positive and uncertain) on medication adherence. ATCS Plus probably slightly (versus control; moderate certainty) or probably (versus usual care; moderate certainty) improves medication adherence but may have little effect on adherence to tests (versus control). IVR probably slightly improves medication adherence versus control (moderate certainty). Compared with usual care, IVR probably improves test adherence and slightly increases medication adherence up to six months but has little or no effect at longer time points (moderate certainty). Unidirectional ATCS, compared with control, may have little effect or slightly improve medication adherence (low certainty). The evidence suggested little or no consistent effect of any ATCS type on clinical outcomes (blood pressure control, blood lipids, asthma control, therapeutic coverage) related to adherence, but only a small number of studies contributed clinical outcome data.The above results focus on areas with the most general findings across conditions. In condition-specific areas, the effects of ATCS varied, including by the type of ATCS intervention in use.Multimodal ATCS probably decrease both cancer pain and chronic pain as well as depression (moderate certainty), but other ATCS types were less effective. Depending on the type of intervention, ATCS may have small effects on outcomes for physical activity, weight management, alcohol consumption, and diabetes mellitus. ATCS have little or no effect on outcomes related to heart failure, hypertension, mental health or smoking cessation, and there is insufficient evidence to determine their effects for preventing alcohol/substance misuse or managing illicit drug addiction, asthma, chronic obstructive pulmonary disease, HIV/AIDS, hypercholesterolaemia, obstructive sleep apnoea, spinal cord dysfunction or psychological stress in carers.Only four trials (3%) reported adverse events, and it was unclear whether these were related to the interventions.

AUTHORS' CONCLUSIONS

ATCS interventions can change patients' health behaviours, improve clinical outcomes and increase healthcare uptake with positive effects in several important areas including immunisation, screening, appointment attendance, and adherence to medications or tests. The decision to integrate ATCS interventions in routine healthcare delivery should reflect variations in the certainty of the evidence available and the size of effects across different conditions, together with the varied nature of ATCS interventions assessed. Future research should investigate both the content of ATCS interventions and the mode of delivery; users' experiences, particularly with regard to acceptability; and clarify which ATCS types are most effective and cost-effective.

Adaptive designs undertaken in clinical research: a review of registered clinical trials

Adaptive designs have the potential to improve efficiency in the evaluation of new medical treatments in comparison to traditional fixed sample size designs. However, they are still not widely used in practice in clinical research. Little research has been conducted to investigate what adaptive designs are being undertaken. This review highlights the current state of registered adaptive designs and their characteristics. The review looked at phase II, II/III and III trials registered on ClinicalTrials.gov from 29 February 2000 to 1 June 2014, supplemented with trials from the National Institute for Health Research register and known adaptive trials. A range of adaptive design search terms were applied to the trials extracted from each database. Characteristics of the adaptive designs were then recorded including funder, therapeutic area and type of adaptation. The results in the paper suggest that the use of adaptive designs has increased. They seem to be most often used in phase II trials and in oncology. In phase III trials, the most popular form of adaptation is the group sequential design. The review failed to capture all trials with adaptive designs, which suggests that the reporting of adaptive designs, such as in clinical trials registers, needs much improving. We recommend that clinical trial registers should contain sections dedicated to the type and scope of the adaptation and that the term 'adaptive design' should be included in the trial title or at least in the brief summary or design sections.

Cross-sector surveys assessing perceptions of key stakeholders towards barriers, concerns and facilitators to the appropriate use of adaptive designs in confirmatory trials

Background

Appropriately conducted adaptive designs (ADs) offer many potential advantages over conventional trials. They make better use of accruing data, potentially saving time, trial participants, and limited resources compared to conventional, fixed sample size designs. However, one can argue that ADs are not implemented as often as they should be, particularly in publicly funded confirmatory trials. This study explored barriers, concerns, and potential facilitators to the appropriate use of ADs in confirmatory trials among key stakeholders.

Methods

We conducted three cross-sectional, online parallel surveys between November 2014 and January 2015. The surveys were based upon findings drawn from in-depth interviews of key research stakeholders, predominantly in the UK, and targeted Clinical Trials Units (CTUs), public funders, and private sector organisations. Response rates were as follows: 30(55 %) UK CTUs, 17(68 %) private sector, and 86(41 %) public funders. A Rating Scale Model was used to rank barriers and concerns in order of perceived importance for prioritisation.

Results

Top-ranked barriers included the lack of bridge funding accessible to UK CTUs to support the design of ADs, limited practical implementation knowledge, preference for traditional mainstream designs, difficulties in marketing ADs to key stakeholders, time constraints to support ADs relative to competing priorities, lack of applied training, and insufficient access to case studies of undertaken ADs to facilitate practical learning and successful implementation. Associated practical complexities and inadequate data management infrastructure to support ADs were reported as more pronounced in the private sector. For funders of public research, the inadequate description of the rationale, scope, and decision-making criteria to guide the planned AD in grant proposals by researchers were all viewed as major obstacles.

Conclusions

There are still persistent and important perceptions of individual and organisational obstacles hampering the use of ADs in confirmatory trials research. Stakeholder perceptions about barriers are largely consistent across sectors, with a few exceptions that reflect differences in organisations’ funding structures, experiences and characterisation of study interventions. Most barriers appear connected to a lack of practical implementation knowledge and applied training, and limited access to case studies to facilitate practical learning.

Electronic supplementary material

The online version of this article (doi:10.1186/s13063-015-1119-x) contains supplementary material, which is available to authorized users.

A theory-based online health behaviour intervention for new university students (U@Uni:LifeGuide): results from a repeat randomized controlled trial

Background

This paper reports the results of a repeat trial assessing the effectiveness of an online theory-based intervention to promote healthy lifestyle behaviours in new university students. The original trial found that the intervention reduced the number of smokers at 6-month follow-up compared with the control condition, but had non-significant effects on the other targeted health behaviours. However, the original trial suffered from low levels of engagement, which the repeat trial sought to rectify.

Methods

Three weeks before staring university, all incoming undergraduate students at a large university in the UK were sent an email inviting them to participate in the study. After completing a baseline questionnaire, participants were randomly allocated to intervention or control conditions. The intervention consisted of a self-affirmation manipulation, health messages based on the theory of planned behaviour and implementation intention tasks. Participants were followed-up 1 and 6 months after starting university. The primary outcome measures were portions of fruit and vegetables consumed, physical activity levels, units of alcohol consumed and smoking status at 6-month follow-up.

Results

The study recruited 2,621 students (intervention n = 1346, control n = 1275), of whom 1495 completed at least one follow-up (intervention n = 696, control n = 799). Intention-to-treat analyses indicated that the intervention had a non-significant effect on the primary outcomes, although the effect of the intervention on fruit and vegetable intake was significant in the per-protocol analyses. Secondary analyses revealed that the intervention had significant effects on having smoked at university (self-report) and on a biochemical marker of alcohol use.

Conclusions

Despite successfully increasing levels of engagement, the intervention did not have a significant effect on the primary outcome measures. The relatively weak effects of the intervention, found in both the original and repeat trials, may be due to the focus on multiple versus single health behaviours. Future interventions targeting the health behaviour of new university students should therefore focus on single health behaviours.

Trial registration

Current Controlled Trials ISRCTN07407344.

Missing steps in a staircase: a qualitative study of the perspectives of key stakeholders on the use of adaptive designs in confirmatory trials

Background

Despite the promising benefits of adaptive designs (ADs), their routine use, especially in confirmatory trials, is lagging behind the prominence given to them in the statistical literature. Much of the previous research to understand barriers and potential facilitators to the use of ADs has been driven from a pharmaceutical drug development perspective, with little focus on trials in the public sector. In this paper, we explore key stakeholders’ experiences, perceptions and views on barriers and facilitators to the use of ADs in publicly funded confirmatory trials.

Methods

Semi-structured, in-depth interviews of key stakeholders in clinical trials research (CTU directors, funding board and panel members, statisticians, regulators, chief investigators, data monitoring committee members and health economists) were conducted through telephone or face-to-face sessions, predominantly in the UK. We purposively selected participants sequentially to optimise maximum variation in views and experiences. We employed the framework approach to analyse the qualitative data.

Results

We interviewed 27 participants. We found some of the perceived barriers to be: lack of knowledge and experience coupled with paucity of case studies, lack of applied training, degree of reluctance to use ADs, lack of bridge funding and time to support design work, lack of statistical expertise, some anxiety about the impact of early trial stopping on researchers’ employment contracts, lack of understanding of acceptable scope of ADs and when ADs are appropriate, and statistical and practical complexities. Reluctance to use ADs seemed to be influenced by: therapeutic area, unfamiliarity, concerns about their robustness in decision-making and acceptability of findings to change practice, perceived complexities and proposed type of AD, among others.

Conclusions

There are still considerable multifaceted, individual and organisational obstacles to be addressed to improve uptake, and successful implementation of ADs when appropriate. Nevertheless, inferred positive change in attitudes and receptiveness towards the appropriate use of ADs by public funders are supportive and are a stepping stone for the future utilisation of ADs by researchers.

Electronic supplementary material

The online version of this article (doi:10.1186/s13063-015-0958-9) contains supplementary material, which is available to authorized users.

Estimating the sample size for a pilot randomised trial to minimise the overall trial sample size for the external pilot and main trial for a continuous outcome variable

Sample size justification is an important consideration when planning a clinical trial, not only for the main trial but also for any preliminary pilot trial. When the outcome is a continuous variable, the sample size calculation requires an accurate estimate of the standard deviation of the outcome measure. A pilot trial can be used to get an estimate of the standard deviation, which could then be used to anticipate what may be observed in the main trial. However, an important consideration is that pilot trials often estimate the standard deviation parameter imprecisely. This paper looks at how we can choose an external pilot trial sample size in order to minimise the sample size of the overall clinical trial programme, that is, the pilot and the main trial together. We produce a method of calculating the optimal solution to the required pilot trial sample size when the standardised effect size for the main trial is known. However, as it may not be possible to know the standardised effect size to be used prior to the pilot trial, approximate rules are also presented. For a main trial designed with 90% power and two-sided 5% significance, we recommend pilot trial sample sizes per treatment arm of 75, 25, 15 and 10 for standardised effect sizes that are extra small (≤0.1), small (0.2), medium (0.5) or large (0.8), respectively.

The disagreeable behaviour of the kappa statistic

It is often of interest to measure the agreement between a number of raters when an outcome is nominal or ordinal. The kappa statistic is used as a measure of agreement. The statistic is highly sensitive to the distribution of the marginal totals and can produce unreliable results. Other statistics such as the proportion of concordance, maximum attainable kappa and prevalence and bias adjusted kappa should be considered to indicate how well the kappa statistic represents agreement in the data. Each kappa should be considered and interpreted based on the context of the data being analysed.

A survey of birth order status of students studying for medical degree at the University of Sheffield

Objective

It is hypothesised that firstborn children and only children are more intelligent with higher intelligence scores having been observed in firstborn or only children. Evidence of the increased intelligence has been suggested by the fact that 21/23 (91%) of US astronauts, 23/43 (53%) of US presidents and between 75 and 80% of students at Harvard are firstborn or only children. It is of interest to investigate, therefore, whether a high achieving career such as medicine has a disproportionate number of firstborn or only children.

Design

A survey of medical students.

Setting

The University of Sheffield Medical School.

Participants

All students studying medicine in the academic year 2011–2012.

Main outcome measures

The proportion of firstborn or only children.

Results

There was a disproportionate number of students who were firstborn or only children: 53% (95% CI 49 to 58%). The expected percentage is 39.8% and therefore we can reject the null hypothesis. The results were consistent across all phases of study.

Conclusions

There is a higher than expected proportion of medical students at the University of Sheffield who are firstborn or only children. The data though highlight the issue of comparing populations. Here we are comparing a population of medical students with a general population. A comparison which may not be appropriate as medical students may be drawn from a subsample of the general population.

A theory-based online health behaviour intervention for new university students (U@Uni): results from a randomised controlled trial

Background

Too few young people engage in behaviours that reduce the risk of morbidity and premature mortality, such as eating healthily, being physically active, drinking sensibly and not smoking. This study sought to assess the efficacy and cost-effectiveness of a theory-based online health behaviour intervention (based on self-affirmation theory, the Theory of Planned Behaviour and implementation intentions) targeting these behaviours in new university students, in comparison to a measurement-only control.

Methods

Two-weeks before starting university all incoming undergraduates at the University of Sheffield were invited to take part in a study of new students’ health behaviour. A randomised controlled design, with a baseline questionnaire, and two follow-ups (1 and 6 months after starting university), was used to evaluate the intervention. Primary outcomes were measures of the four health behaviours targeted by the intervention at 6-month follow-up, i.e., portions of fruit and vegetables, metabolic equivalent of tasks (physical activity), units of alcohol, and smoking status.

Results

The study recruited 1,445 students (intervention n = 736, control n = 709, 58% female, Mean age = 18.9 years), of whom 1,107 completed at least one follow-up (23% attrition). The intervention had a statistically significant effect on one primary outcome, smoking status at 6-month follow-up, with fewer smokers in the intervention arm (8.7%) than in the control arm (13.0%; Odds ratio = 1.92, p = .010). There were no significant intervention effects on the other primary outcomes (physical activity, alcohol or fruit and vegetable consumption) at 6-month follow-up.

Conclusions

The results of the RCT indicate that the online health behaviour intervention reduced smoking rates, but it had little effect on fruit and vegetable intake, physical activity or alcohol consumption, during the first six months at university. However, engagement with the intervention was low. Further research is needed before strong conclusions can be made regarding the likely effectiveness of the intervention to promote health lifestyle habits in new university students.

Trial registration

Current Controlled Trials, ISRCTN67684181.

The statistical interpretation of pilot trials: should significance thresholds be reconsidered?

BACKGROUND

In an evaluation of a new health technology, a pilot trial may be undertaken prior to a trial that makes a definitive assessment of benefit. The objective of pilot studies is to provide sufficient evidence that a larger definitive trial can be undertaken and, at times, to provide a preliminary assessment of benefit.

METHODS

We describe significance thresholds, confidence intervals and surrogate markers in the context of pilot studies and how Bayesian methods can be used in pilot trials. We use a worked example to illustrate the issues raised.

RESULTS

We show how significance levels other than the traditional 5% should be considered to provide preliminary evidence for efficacy and how estimation and confidence intervals should be the focus to provide an estimated range of possible treatment effects. We also illustrate how Bayesian methods could also assist in the early assessment of a health technology.

CONCLUSIONS

We recommend that in pilot trials the focus should be on descriptive statistics and estimation, using confidence intervals, rather than formal hypothesis testing and that confidence intervals other than 95% confidence intervals, such as 85% or 75%, be used for the estimation. The confidence interval should then be interpreted with regards to the minimum clinically important difference. We also recommend that Bayesian methods be used to assist in the interpretation of pilot trials. Surrogate endpoints can also be used in pilot trials but they must reliably predict the overall effect on the clinical outcome.

An investigation of the impact of futility analysis in publicly funded trials

Background

Publicly funded trials regularly fail to recruit their target sample size or find a significant positive result. Adaptive clinical trials which may partly mediate against the problems are not often applied. In this paper we investigate the potential of a form of adaption in a clinical trial - a futility analysis - to see if it has potential to improve publicly funded trials.

Methods

Outcome data from trials funded by two UK bodies, the Health Technology Assessment (HTA) programme and the UK Medical Research Council (MRC), were collected. These data were then used to simulate each trial with a single futility analysis using conditional power, undertaken after 50% to 90% of the patients had been recruited. Thirty-three trials recruiting between 2002 and 2008 met the inclusion criteria. Stopping boundaries of conditional powers of 20%, 30% and 40% were considered and outcomes included the number of trials successfully stopped and number of patients saved.

Results

Inclusion of a futility analysis after 75% of the patients had been recruited would have potentially resulted in 10 trials, which went on to have negative results, correctly stopping for futility using a stopping boundary of 30%. A total of 807 patients across all the trials would potentially have been saved using these futility parameters. The proportion of studies successfully recruiting would also have increased from 45% to 64%.

Conclusions

A futility assessment has the potential to increase efficiency, save patients and decrease costs in publicly funded trials. While there are logistical issues in undertaking futility assessments we recommend that investigators should aim to include a futility analysis in their trial design wherever possible.

Preventing and lessening exacerbations of asthma in school-age children associated with a new term (PLEASANT): study protocol for a cluster randomised control trial

BACKGROUND

Within the UK, during September, there is a pronounced increase in the number of unscheduled medical contacts by school-aged children (4-16 years) with asthma. It is thought that that this might be caused by the return back to school after the summer holidays, suddenly mixing with other children again and picking up viruses which could affect their asthma. There is also a drop in the number of prescriptions administered in August. It is possible therefore that children might not be taking their medication as they should during the summer contributing to them becoming ill when they return to school.It is hoped that a simple intervention from the GP to parents of children with asthma at the start of the summer holiday period, highlighting the importance of maintaining asthma medication can help prevent increased asthma exacerbation, and unscheduled NHS appointments, following return to school in September.

METHODS/DESIGN

PLEASANT is a cluster randomised trial. A total of 140 General Practices (GPs) will be recruited into the trial; 70 GPs randomised to the intervention and 70 control practices of "usual care". An average practice is expected to have approximately 100 children (aged 4-16 with a diagnosis of asthma) hence observational data will be collected on around 14000 children over a 24-month period. The Clinical Practice Research Datalink will collect all data required for the study which includes diagnostic, prescription and referral data.

DISCUSSION

The trial will assess whether the intervention can reduce exacerbation of asthma and unscheduled medical contacts in school-aged children associated with the return to school after the summer holidays. It has the potential to benefit the health and quality of life of children with asthma while also improving the effectiveness of NHS services by reducing NHS use in one of the busiest months of the year.An exploratory health economic analysis will gauge any cost saving associated with the intervention and subsequent impacts on quality of life. If results for the intervention are positive it is hoped that this could be adopted as part of routine care management of childhood asthma in general practice.

TRIAL REGISTRATION

Current controlled trials: ISRCTN03000938 (assigned 19/10/12) http://www.controlled-trials.com/ISRCTN03000938/.

UKCRN ID

13572.