Choosing the target difference ('effect size') for a randomised controlled trial - DELTA2 guidance protocol

Sample size calculations in high-profile surgical trials that use patient-reported outcome measures: systematic review

Sample size calculations from high-profile surgical RCTs that used a patient-reported outcome measure as primary outcome were reviewed systematically against Difference ELicitation in TriAls (DELTA2) standards, with a focus on target differences. In this sample of trials, there was frequent use of suboptimal methods to determine the target difference, and sample size calculations were generally not reported to DELTA2 standards. This risks over-recruitment and/or erroneous trial conclusions, which clinicians should be aware of when interpreting published trials.

Minimal clinically important differences in randomised clinical trials on pain management after total hip and knee arthroplasty: a systematic review

BACKGROUND

Sample size determination is essential for reliable hypothesis testing in clinical trials and should rely on adequate sample size calculations with alpha, beta, variance, and an effect size being the minimal clinically important difference (MCID). This facilitates interpretation of the clinical relevance of statistically significant results. No gold standard for MCIDs exists in postoperative pain research.

METHODS

We searched Cochrane Central Register of Controlled Trials, MEDLINE, and Embase for English language articles on randomised trials investigating analgesic interventions after total hip or knee arthroplasty. Primary outcomes were the reported MCIDs for pain score and cumulated rescue opioid consumption. Secondary outcomes included reported sample size calculations and propensity to report statistical significance without reaching MCID. Trend analyses were conducted using statistical process control.

RESULTS

We included 570 trials. Median MCID for 0-24 h opioid consumption was 10 mg i.v. morphine equivalents for absolute reductions (interquartile range [IQR]: 6.8-14.5) and relative 40% (IQR: 30-50%). Median MCIDs for pain scores were absolute 15 mm at rest (IQR: 10-20) and 18 mm during movement (IQR: 10-20) on a 0-100 mm VAS and relative 30% (IQR: 20-30%). No trends were demonstrated for MCIDs. Adequate sample size calculations were reported in 34% of trials. In 46% of trials with statistically significant primary outcomes, the differences did not reach the predetermined MCID.

CONCLUSIONS

We provide clinician-perceived MCID estimates for rescue opioid consumption and pain scores that can be used for sample size calculations until reliable evidence-based patient-rated MCIDs emerge. Nearly half of the trials with significant findings did not reach the predetermined MCID.

Using Time Trade-Off Methods to Elicit Short-Term Utilities Associated with Treatments for Bulbar Urethral Stricture

BACKGROUND

Recurrent urethral stricture is usually treated with either open urethroplasty or endoscopic urethrotomy. Both of the procedures cause short-term utility loss, which may not be captured by standard utility questionnaires due to the challenges of completing a standard instrument at the time of an acute episode of short duration, especially within a clinical trial setting. We propose to use time trade-off (TTO) methods to estimate these short-term utility losses.

OBJECTIVE

The aim was to compare the use of two alternative TTO methods to elicit patients' short-term utilities following surgical treatments for recurrent urethral stricture.

METHOD

Two variants of TTO (chained and conventional) were used. Six health profiles were developed-three for each procedure. Forty participants took part, with 20 randomly allocated to each TTO method.

RESULTS

Thirty-eight participants provided usable data for analysis. Estimated utility values decreased as the severity of the health profiles increased. There was no evidence that utility values differed between elicitation methods or procedures for mild {ranging from 0.79 (standard deviation [SD] 0.17) to 0.83 [SD 0.20]} and moderate (ranging from 0.54 [SD 0.24] to 0.67 [SD 0.21]) health states, although they appeared to differ for severe health states (ranging from 0.29 [SD 0.20] to 0.56 [SD 0.24]).

CONCLUSION

The study demonstrates the feasibility and value of eliciting patients' short-term utilities. Given the small sample size, the study findings are tentative. Further research with a larger sample size is needed to determine the appropriate TTO method to use and how the elicited utilities can be used in combination with standard cost-utility assessments to aid decision making.

New guidance to improve sample size calculations for trials: eliciting the target difference

BACKGROUND

Sample size calculations are central to the design of health research trials. To ensure that the trial provides good evidence to answer the trial's research question, the target effect size (difference in means or proportions, odds ratio, relative risk or hazard ratio between trial arms) must be specified under the conventional approach to determining the sample size. However, until now, there has not been comprehensive guidance on how to specify this effect.

MAIN TEXT

This is a commentary on a collection of papers from two important projects, DELTA (Difference ELicitation in TriAls) and DELTA2 that aim to provide evidence-based guidance on systematically determining the target effect size, or difference and the resultant sample sizes for trials. In addition to surveying methods that researchers are using in practice, the research team met with various experts (statisticians, methodologists, clinicians and funders); reviewed guidelines from funding agencies; and reviewed recent methodological literature. The DELTA2 guidance stresses specifying important and realistic differences, and undertaking sensitivity analyses in calculating sample sizes. It gives recommendations on how to find appropriate differences, conduct the sample size calculation(s) and how to report these in grant applications, protocols and manuscripts. It is hoped that this will contribute not only to better powered studies, but better reporting and reproducibility and thinking about what a trial should be designed to achieve.

CONCLUSIONS

The DELTA researchers have developed a set of comprehensive guidance documents that are welcome and will almost certainly improve the way that trials are designed and reported.

Choosing the target difference and undertaking and reporting the sample size calculation for a randomised controlled trial - the development of the DELTA2 guidance

BACKGROUND

A key step in the design of a randomised controlled trial is the estimation of the number of participants needed. The most common approach is to specify a target difference in the primary outcome between the randomised groups and then estimate the corresponding sample size. The sample size is chosen to provide reassurance that the trial will have high statistical power to detect the target difference at the planned statistical significance level. Alternative approaches are also available, though most still require specification of a target difference. The sample size has many implications for the conduct of the study, as well as incurring scientific and ethical aspects. Despite the critical role of the target difference for the primary outcome in the design of a randomised controlled trial (RCT), the manner in which it is determined has received little attention. This article reports the development of the DELTA² guidance on the specification and reporting of the target difference for the primary outcome in a sample size calculation for a RCT.

METHODS

The DELTA² (Difference ELicitation in TriAls) project has five components comprising systematic literature reviews of recent methodological developments (stage 1) and existing funder guidance (stage 2), a Delphi study (stage 3), a 2-day consensus meeting bringing together researchers, funders and patient representatives (stage 4), and the preparation and dissemination of a guidance document (stage 5).

RESULTS

The project started in April 2016. The literature search identified 28 articles of methodological developments relevant to a method for specifying a target difference. A Delphi study involving 69 participants, along with a 2-day consensus meeting were conducted. In addition, further engagement sessions were held at two international conferences. The main guidance text was finalised on April 18, 2018, after revision informed by feedback gathered from stages 2 and 3 and from funder representatives.

DISCUSSION

The DELTA² Delphi study identified a number of areas (such as practical recommendations and examples, greater coverage of different trial designs and statistical approaches) of particular interest amongst stakeholders which new guidance was desired to meet. New relevant references were identified by the review. Such findings influenced the scope, drafting and revision of the guidance. While not all suggestions could be accommodated, it is hoped that the process has led to a more useful and practical document.

A study of target effect sizes in randomised controlled trials published in the Health Technology Assessment journal

BACKGROUND

When designing a randomised controlled trial (RCT), an important consideration is the sample size required. This is calculated from several components; one of which is the target difference. This study aims to review the currently reported methods of elicitation of the target difference as well as to quantify the target differences used in Health Technology Assessment (HTA)-funded trials.

METHODS

Trials were identified from the National Institute of Health Research Health Technology Assessment journal. A total of 177 RCTs published between 2006 and 2016 were assessed for eligibility. Eligibility was established by the design of the trial and the quality of data available. The trial designs were parallel-group, superiority RCTs with a continuous primary endpoint. Data were extracted and the standardised anticipated and observed effect size estimates were calculated. Exclusion criteria was based on trials not providing enough detail in the sample size calculation and results, and trials not being of parallel-group, superiority design.

RESULTS

A total of 107 RCTs were included in the study from 102 reports. The most commonly reported method for effect size derivation was a review of evidence and use of previous research (52.3%). This was common across all clinical areas. The median standardised target effect size was 0.30 (interquartile range: 0.20-0.38), with the median standardised observed effect size 0.11 (IQR 0.05-0.29). The maximum anticipated and observed effect sizes were 0.76 and 1.18, respectively. Only two trials had anticipated target values above 0.60.

CONCLUSION

The most commonly reported method of elicitation of the target effect size is previous published research. The average target effect size was 0.3. A clear distinction between the target difference and the minimum clinically important difference is recommended when designing a trial. Transparent explanation of target difference elicitation is advised, with multiple methods including a review of evidence and opinion-seeking advised as the more optimal methods for effect size quantification.

Can testing clinical significance reduce false positive rates in randomized controlled trials? A snap review

OBJECTIVE

The use of minimum clinically important difference in the hypothesis formulation for superiority trials is similar in principle to the concept of non-inferiority or equivalence trial. However, most clinical trials are analysed testing zero clinical difference. Since the minimum clinically important difference is pre-defined for power calculation, it is important to incorporate it in both the hypothesis testing and the interpretation of findings from clinical trials.

RESULTS

We reviewed a set of 50 publications (25 with binary outcome, and 25 with survival time outcome). 20% of the 50 published trials that were statistically significant, were also clinically significant based on the minimum clinically important risk differences (or hazard ratio) used for their power calculations. This snap review seems to suggest that most published trials with statistically significant results were less likely to be clinically significant, which may partly explain the high false positive findings associated with findings from superiority trials. Furthermore, none of the reviewed publications explicitly used minimum clinically important difference in the interpretation of their findings. However, a systematic review is needed to critically appraise the impact of the current practice on false positive rate in published trials with significant findings.